{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 PyTorch第一步\n",
    "\n",
    "PyTorch的简洁设计使得它入门很简单，在深入介绍PyTorch之前，本节将先介绍一些PyTorch的基础知识，使得读者能够对PyTorch有一个大致的了解，并能够用PyTorch搭建一个简单的神经网络。部分内容读者可能暂时不太理解，可先不予以深究，本书的第3章和第4章将会对此进行深入讲解。\n",
    "\n",
    "本节内容参考了PyTorch官方教程[^1]并做了相应的增删修改，使得内容更贴合新版本的PyTorch接口，同时也更适合新手快速入门。另外本书需要读者先掌握基础的Numpy使用，其他相关知识推荐读者参考CS231n的教程[^2]。\n",
    "\n",
    "[^1]: http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html\n",
    "[^2]: http://cs231n.github.io/python-numpy-tutorial/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensor\n",
    "\n",
    "Tensor是PyTorch中重要的数据结构，可认为是一个高维数组。它可以是一个数（标量）、一维数组（向量）、二维数组（矩阵）以及更高维的数组。Tensor和Numpy的ndarrays类似，但Tensor可以使用GPU进行加速。Tensor的使用和Numpy及Matlab的接口十分相似，下面通过几个例子来看看Tensor的基本使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    " from __future__ import print_function\n",
    "import torch as t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "1.00000e-34 *\n",
       "  0.0000  0.0000  3.1803\n",
       "  0.0000  0.0002  0.0000\n",
       "  0.0002  0.0000  0.0002\n",
       "  0.0000  3.2308  0.0000\n",
       "  0.0000  0.0000  0.0000\n",
       "[torch.FloatTensor of size 5x3]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 构建 5x3 矩阵，只是分配了空间，未初始化\n",
    "x = t.Tensor(5, 3)  \n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.1613  0.8384  0.9574\n",
       " 0.8003  0.7798  0.6283\n",
       " 0.0428  0.0455  0.1856\n",
       " 0.6404  0.0198  0.4298\n",
       " 0.7187  0.8979  0.3566\n",
       "[torch.FloatTensor of size 5x3]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用[0,1]均匀分布随机初始化二维数组\n",
    "x = t.rand(5, 3)  \n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([5, 3])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(3, 3)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(x.size()) # 查看x的形状\n",
    "x.size()[1], x.size(1) # 查看列的个数, 两种写法等价"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`torch.Size` 是tuple对象的子类，因此它支持tuple的所有操作，如x.size()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.7069  1.1960  1.5466\n",
       " 1.7804  1.7523  1.2713\n",
       " 0.3116  0.6676  1.0148\n",
       " 1.4244  0.1380  1.3406\n",
       " 1.1767  1.1861  0.5253\n",
       "[torch.FloatTensor of size 5x3]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = t.rand(5, 3)\n",
    "# 加法的第一种写法\n",
    "x + y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.7069  1.1960  1.5466\n",
       " 1.7804  1.7523  1.2713\n",
       " 0.3116  0.6676  1.0148\n",
       " 1.4244  0.1380  1.3406\n",
       " 1.1767  1.1861  0.5253\n",
       "[torch.FloatTensor of size 5x3]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 加法的第二种写法\n",
    "t.add(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.7069  1.1960  1.5466\n",
       " 1.7804  1.7523  1.2713\n",
       " 0.3116  0.6676  1.0148\n",
       " 1.4244  0.1380  1.3406\n",
       " 1.1767  1.1861  0.5253\n",
       "[torch.FloatTensor of size 5x3]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 加法的第三种写法：指定加法结果的输出目标为result\n",
    "result = t.Tensor(5, 3) # 预先分配空间\n",
    "t.add(x, y, out=result) # 输入到result\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "最初y\n",
      "\n",
      " 0.5456  0.3576  0.5892\n",
      " 0.9800  0.9725  0.6430\n",
      " 0.2688  0.6221  0.8293\n",
      " 0.7839  0.1182  0.9108\n",
      " 0.4580  0.2881  0.1688\n",
      "[torch.FloatTensor of size 5x3]\n",
      "\n",
      "第一种加法，y的结果\n",
      "\n",
      " 0.5456  0.3576  0.5892\n",
      " 0.9800  0.9725  0.6430\n",
      " 0.2688  0.6221  0.8293\n",
      " 0.7839  0.1182  0.9108\n",
      " 0.4580  0.2881  0.1688\n",
      "[torch.FloatTensor of size 5x3]\n",
      "\n",
      "第二种加法，y的结果\n",
      "\n",
      " 0.7069  1.1960  1.5466\n",
      " 1.7804  1.7523  1.2713\n",
      " 0.3116  0.6676  1.0148\n",
      " 1.4244  0.1380  1.3406\n",
      " 1.1767  1.1861  0.5253\n",
      "[torch.FloatTensor of size 5x3]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('最初y')\n",
    "print(y)\n",
    "\n",
    "print('第一种加法，y的结果')\n",
    "y.add(x) # 普通加法，不改变y的内容\n",
    "print(y)\n",
    "\n",
    "print('第二种加法，y的结果')\n",
    "y.add_(x) # inplace 加法，y变了\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意，函数名后面带下划线**`_`** 的函数会修改Tensor本身。例如，`x.add_(y)`和`x.t_()`会改变 `x`，但`x.add(y)`和`x.t()`返回一个新的Tensor， 而`x`不变。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.8384\n",
       " 0.7798\n",
       " 0.0455\n",
       " 0.0198\n",
       " 0.8979\n",
       "[torch.FloatTensor of size 5]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Tensor的选取操作与Numpy类似\n",
    "x[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor还支持很多操作，包括数学运算、线性代数、选择、切片等等，其接口设计与Numpy极为相似。更详细的使用方法，会在第三章系统讲解。\n",
    "\n",
    "Tensor和Numpy的数组之间的互操作非常容易且快速。对于Tensor不支持的操作，可以先转为Numpy数组处理，之后再转回Tensor。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       "[torch.FloatTensor of size 5]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.ones(5) # 新建一个全1的Tensor\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.,  1.,  1.,  1.,  1.], dtype=float32)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.numpy() # Tensor -> Numpy\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.  1.  1.  1.  1.]\n",
      "\n",
      " 1\n",
      " 1\n",
      " 1\n",
      " 1\n",
      " 1\n",
      "[torch.DoubleTensor of size 5]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "a = np.ones(5)\n",
    "b = t.from_numpy(a) # Numpy->Tensor\n",
    "print(a)\n",
    "print(b) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor和numpy对象共享内存，所以他们之间的转换很快，而且几乎不会消耗什么资源。但这也意味着，如果其中一个变了，另外一个也会随之改变。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 2.  2.  2.  2.  2.]\n",
      "\n",
      " 2\n",
      " 2\n",
      " 2\n",
      " 2\n",
      " 2\n",
      "[torch.DoubleTensor of size 5]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "b.add_(1) # 以`_`结尾的函数会修改自身\n",
    "print(a)\n",
    "print(b) # Tensor和Numpy共享内存"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor可通过`.cuda` 方法转为GPU的Tensor，从而享受GPU带来的加速运算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在不支持CUDA的机器下，下一步不会运行\n",
    "if t.cuda.is_available():\n",
    "    x = x.cuda()\n",
    "    y = y.cuda()\n",
    "    x + y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此处可能发现GPU运算的速度并未提升太多，这是因为x和y太小且运算也较为简单，而且将数据从内存转移到显存还需要花费额外的开销。GPU的优势需在大规模数据和复杂运算下才能体现出来。\n",
    "\n",
    "### Autograd: 自动微分\n",
    "\n",
    "深度学习的算法本质上是通过反向传播求导数，而PyTorch的**`Autograd`**模块则实现了此功能。在Tensor上的所有操作，Autograd都能为它们自动提供微分，避免了手动计算导数的复杂过程。\n",
    " \n",
    "`autograd.Variable`是Autograd中的核心类，它简单封装了Tensor，并支持几乎所有Tensor有的操作。Tensor在被封装为Variable之后，可以调用它的`.backward`实现反向传播，自动计算所有梯度。Variable的数据结构如图2-6所示。\n",
    "\n",
    "\n",
    "![图2-6:Variable的数据结构](imgs/autograd_Variable.svg)\n",
    "\n",
    "\n",
    "Variable主要包含三个属性。\n",
    "- `data`：保存Variable所包含的Tensor\n",
    "- `grad`：保存`data`对应的梯度，`grad`也是个Variable，而不是Tensor，它和`data`的形状一样。\n",
    "- `grad_fn`：指向一个`Function`对象，这个`Function`用来反向传播计算输入的梯度，具体细节会在下一章讲解。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.autograd import Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable containing:\n",
       " 1  1\n",
       " 1  1\n",
       "[torch.FloatTensor of size 2x2]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用Tensor新建一个Variable\n",
    "x = Variable(t.ones(2, 2), requires_grad = True)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable containing:\n",
       " 4\n",
       "[torch.FloatTensor of size 1]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = x.sum()\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch.autograd.function.SumBackward at 0x7fc1e402de58>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.grad_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.backward() # 反向传播,计算梯度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable containing:\n",
       " 1  1\n",
       " 1  1\n",
       "[torch.FloatTensor of size 2x2]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# y = x.sum() = (x[0][0] + x[0][1] + x[1][0] + x[1][1])\n",
    "# 每个值的梯度都为1\n",
    "x.grad "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意：`grad`在反向传播过程中是累加的(accumulated)，这意味着每一次运行反向传播，梯度都会累加之前的梯度，所以反向传播之前需把梯度清零。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable containing:\n",
       " 2  2\n",
       " 2  2\n",
       "[torch.FloatTensor of size 2x2]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable containing:\n",
       " 3  3\n",
       " 3  3\n",
       "[torch.FloatTensor of size 2x2]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0  0\n",
       " 0  0\n",
       "[torch.FloatTensor of size 2x2]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 以下划线结束的函数是inplace操作，就像add_\n",
    "x.grad.data.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable containing:\n",
       " 1  1\n",
       " 1  1\n",
       "[torch.FloatTensor of size 2x2]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Variable和Tensor具有近乎一致的接口，在实际使用中可以无缝切换。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 0.5403  0.5403  0.5403  0.5403  0.5403\n",
      " 0.5403  0.5403  0.5403  0.5403  0.5403\n",
      " 0.5403  0.5403  0.5403  0.5403  0.5403\n",
      " 0.5403  0.5403  0.5403  0.5403  0.5403\n",
      "[torch.FloatTensor of size 4x5]\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.5403  0.5403  0.5403  0.5403  0.5403\n",
       " 0.5403  0.5403  0.5403  0.5403  0.5403\n",
       " 0.5403  0.5403  0.5403  0.5403  0.5403\n",
       " 0.5403  0.5403  0.5403  0.5403  0.5403\n",
       "[torch.FloatTensor of size 4x5]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = Variable(t.ones(4,5))\n",
    "y = t.cos(x)\n",
    "x_tensor_cos = t.cos(x.data)\n",
    "print(y)\n",
    "x_tensor_cos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  神经网络\n",
    "\n",
    "Autograd实现了反向传播功能，但是直接用来写深度学习的代码在很多情况下还是稍显复杂，torch.nn是专门为神经网络设计的模块化接口。nn构建于 Autograd之上，可用来定义和运行神经网络。nn.Module是nn中最重要的类，可把它看成是一个网络的封装，包含网络各层定义以及forward方法，调用forward(input)方法，可返回前向传播的结果。下面就以最早的卷积神经网络：LeNet为例，来看看如何用`nn.Module`实现。LeNet的网络结构如图2-7所示。\n",
    "\n",
    "![图2-7:LeNet网络结构](imgs/nn_lenet.png)\n",
    "\n",
    "这是一个基础的前向传播(feed-forward)网络: 接收输入，经过层层传递运算，得到输出。\n",
    "\n",
    "#### 定义网络\n",
    "\n",
    "定义网络时，需要继承`nn.Module`，并实现它的forward方法，把网络中具有可学习参数的层放在构造函数`__init__`中。如果某一层(如ReLU)不具有可学习的参数，则既可以放在构造函数中，也可以不放，但建议不放在其中，而在forward中使用`nn.functional`代替。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net (\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear (400 -> 120)\n",
      "  (fc2): Linear (120 -> 84)\n",
      "  (fc3): Linear (84 -> 10)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        # nn.Module子类的函数必须在构造函数中执行父类的构造函数\n",
    "        # 下式等价于nn.Module.__init__(self)\n",
    "        super(Net, self).__init__()\n",
    "        \n",
    "        # 卷积层 '1'表示输入图片为单通道, '6'表示输出通道数，'5'表示卷积核为5*5\n",
    "        self.conv1 = nn.Conv2d(1, 6, 5) \n",
    "        # 卷积层\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5) \n",
    "        # 仿射层/全连接层，y = Wx + b\n",
    "        self.fc1   = nn.Linear(16*5*5, 120) \n",
    "        self.fc2   = nn.Linear(120, 84)\n",
    "        self.fc3   = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x): \n",
    "        # 卷积 -> 激活 -> 池化 \n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2) \n",
    "        # reshape，‘-1’表示自适应\n",
    "        x = x.view(x.size()[0], -1) \n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)        \n",
    "        return x\n",
    "\n",
    "net = Net()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "只要在nn.Module的子类中定义了forward函数，backward函数就会自动被实现(利用`Autograd`)。在`forward` 函数中可使用任何Variable支持的函数，还可以使用if、for循环、print、log等Python语法，写法和标准的Python写法一致。\n",
    "\n",
    "网络的可学习参数通过`net.parameters()`返回，`net.named_parameters`可同时返回可学习的参数及名称。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n"
     ]
    }
   ],
   "source": [
    "params = list(net.parameters())\n",
    "print(len(params))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv1.weight : torch.Size([6, 1, 5, 5])\n",
      "conv1.bias : torch.Size([6])\n",
      "conv2.weight : torch.Size([16, 6, 5, 5])\n",
      "conv2.bias : torch.Size([16])\n",
      "fc1.weight : torch.Size([120, 400])\n",
      "fc1.bias : torch.Size([120])\n",
      "fc2.weight : torch.Size([84, 120])\n",
      "fc2.bias : torch.Size([84])\n",
      "fc3.weight : torch.Size([10, 84])\n",
      "fc3.bias : torch.Size([10])\n"
     ]
    }
   ],
   "source": [
    "for name,parameters in net.named_parameters():\n",
    "    print(name,':',parameters.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "forward函数的输入和输出都是Variable，只有Variable才具有自动求导功能，而Tensor是没有的，所以在输入时，需把Tensor封装成Variable。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 10])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input = Variable(t.randn(1, 1, 32, 32))\n",
    "out = net(input)\n",
    "out.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "net.zero_grad() # 所有参数的梯度清零\n",
    "out.backward(Variable(t.ones(1,10))) # 反向传播"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要注意的是，torch.nn只支持mini-batches，不支持一次只输入一个样本，即一次必须是一个batch。但如果只想输入一个样本，则用 `input.unsqueeze(0)`将batch_size设为１。例如 `nn.Conv2d` 输入必须是4维的，形如$nSamples \\times nChannels \\times Height \\times Width$。可将nSample设为1，即$1 \\times nChannels \\times Height \\times Width$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 损失函数\n",
    "\n",
    "nn实现了神经网络中大多数的损失函数，例如nn.MSELoss用来计算均方误差，nn.CrossEntropyLoss用来计算交叉熵损失。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable containing:\n",
       " 28.4127\n",
       "[torch.FloatTensor of size 1]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = net(input)\n",
    "target = Variable(t.arange(0,10))  \n",
    "criterion = nn.MSELoss()\n",
    "loss = criterion(output, target)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果对loss进行反向传播溯源(使用`gradfn`属性)，可看到它的计算图如下：\n",
    "\n",
    "```\n",
    "input -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d  \n",
    "      -> view -> linear -> relu -> linear -> relu -> linear \n",
    "      -> MSELoss\n",
    "      -> loss\n",
    "```\n",
    "\n",
    "当调用`loss.backward()`时，该图会动态生成并自动微分，也即会自动计算图中参数(Parameter)的导数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "反向传播之前 conv1.bias的梯度\n",
      "Variable containing:\n",
      " 0\n",
      " 0\n",
      " 0\n",
      " 0\n",
      " 0\n",
      " 0\n",
      "[torch.FloatTensor of size 6]\n",
      "\n",
      "反向传播之后 conv1.bias的梯度\n",
      "Variable containing:\n",
      "-0.0652\n",
      "-0.0077\n",
      " 0.0081\n",
      "-0.1228\n",
      " 0.0249\n",
      " 0.0587\n",
      "[torch.FloatTensor of size 6]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 运行.backward，观察调用之前和调用之后的grad\n",
    "net.zero_grad() # 把net中所有可学习参数的梯度清零\n",
    "print('反向传播之前 conv1.bias的梯度')\n",
    "print(net.conv1.bias.grad)\n",
    "loss.backward()\n",
    "print('反向传播之后 conv1.bias的梯度')\n",
    "print(net.conv1.bias.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 优化器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在反向传播计算完所有参数的梯度后，还需要使用优化方法来更新网络的权重和参数，例如随机梯度下降法(SGD)的更新策略如下：\n",
    "```\n",
    "weight = weight - learning_rate * gradient\n",
    "```\n",
    "\n",
    "手动实现如下：\n",
    "\n",
    "```python\n",
    "learning_rate = 0.01\n",
    "for f in net.parameters():\n",
    "    f.data.sub_(f.grad.data * learning_rate)# inplace 减法\n",
    "```\n",
    "\n",
    "`torch.optim`中实现了深度学习中绝大多数的优化方法，例如RMSProp、Adam、SGD等，更便于使用，因此大多数时候并不需要手动写上述代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "#新建一个优化器，指定要调整的参数和学习率\n",
    "optimizer = optim.SGD(net.parameters(), lr = 0.01)\n",
    "\n",
    "# 在训练过程中\n",
    "# 先梯度清零(与net.zero_grad()效果一样)\n",
    "optimizer.zero_grad() \n",
    "\n",
    "# 计算损失\n",
    "output = net(input)\n",
    "loss = criterion(output, target)\n",
    "\n",
    "#反向传播\n",
    "loss.backward()\n",
    "\n",
    "#更新参数\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "####  数据加载与预处理\n",
    "\n",
    "在深度学习中数据加载及预处理是非常复杂繁琐的，但PyTorch提供了一些可极大简化和加快数据处理流程的工具。同时，对于常用的数据集，PyTorch也提供了封装好的接口供用户快速调用，这些数据集主要保存在torchvison中。\n",
    "\n",
    "`torchvision`实现了常用的图像数据加载功能，例如Imagenet、CIFAR10、MNIST等，以及常用的数据转换操作，这极大地方便了数据加载，并且代码具有可重用性。\n",
    "\n",
    "\n",
    "### 小试牛刀：CIFAR-10分类\n",
    "\n",
    "下面我们来尝试实现对CIFAR-10数据集的分类，步骤如下: \n",
    "\n",
    "1. 使用torchvision加载并预处理CIFAR-10数据集\n",
    "2. 定义网络\n",
    "3. 定义损失函数和优化器\n",
    "4. 训练网络并更新网络参数\n",
    "5. 测试网络\n",
    "\n",
    "####   CIFAR-10数据加载及预处理\n",
    "\n",
    "CIFAR-10[^3]是一个常用的彩色图片数据集，它有10个类别: 'airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'。每张图片都是$3\\times32\\times32$，也即3-通道彩色图片，分辨率为$32\\times32$。\n",
    "\n",
    "[^3]: http://www.cs.toronto.edu/~kriz/cifar.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision as tv\n",
    "import torchvision.transforms as transforms\n",
    "from torchvision.transforms import ToPILImage\n",
    "show = ToPILImage() # 可以把Tensor转成Image，方便可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "# 第一次运行程序torchvision会自动下载CIFAR-10数据集，\n",
    "# 大约100M，需花费一定的时间，\n",
    "# 如果已经下载有CIFAR-10，可通过root参数指定\n",
    "\n",
    "# 定义对数据的预处理\n",
    "transform = transforms.Compose([\n",
    "        transforms.ToTensor(), # 转为Tensor\n",
    "        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)), # 归一化\n",
    "                             ])\n",
    "\n",
    "# 训练集\n",
    "trainset = tv.datasets.CIFAR10(\n",
    "                    root='/home/cy/data/', \n",
    "                    train=True, \n",
    "                    download=True,\n",
    "                    transform=transform)\n",
    "\n",
    "trainloader = t.utils.data.DataLoader(\n",
    "                    trainset, \n",
    "                    batch_size=4,\n",
    "                    shuffle=True, \n",
    "                    num_workers=2)\n",
    "\n",
    "# 测试集\n",
    "testset = tv.datasets.CIFAR10(\n",
    "                    '/home/cy/data/',\n",
    "                    train=False, \n",
    "                    download=True, \n",
    "                    transform=transform)\n",
    "\n",
    "testloader = t.utils.data.DataLoader(\n",
    "                    testset,\n",
    "                    batch_size=4, \n",
    "                    shuffle=False,\n",
    "                    num_workers=2)\n",
    "\n",
    "classes = ('plane', 'car', 'bird', 'cat',\n",
    "           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset对象是一个数据集，可以按下标访问，返回形如(data, label)的数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ship\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAIAAAD/gAIDAAALVElEQVR4nO1cW3MVxxGe2d1zk46E\nhEASlpCEBaEo43K5UqmUKz8jpCo/MQ/Jj0j5JQllLGIw2NxsK4iLERJH17PXPEz316OZxdLoeb4X\nWruzvbPDfNOX6Tn64cuRUkopVde1OomqEbmsaqcZhIKbFTVJVVV5jRsWRGdRlaRc8d2Gbmtu3/AD\nTdM4Ql4m0tXavYs+NI1mVepjX9pUckUXlXMX7RMVcWbEwQpAppkCEACttMgsJiw1uOK1EYHvJWht\nvQWqUhY0s0FrJqbGYy5V00S6Bwjf5RpdSZKU/vaorRrpldau2oRfFGdWAOJgBSBLZMLy5OS/Ey1D\nCakRXqAZBDZJunEayRVrkguNEpe3iSwO3DkxWCCv9R0ed1J0hvsO9uFtYLTSNg3d5QhsjTMrAHGw\nApBZfHJnvj2zwYtaYTLz5GzQ5qQiS2w8U6tsHmm3WeL1wmK9e+VEJ+C7ijlkN5VvZQluJM5HqTbv\nF0Y6zqwAxMEKQBysAGRwWH0XO7GMqCxVWNr4AawFvivQ1O6CaLvdWpNLXXNEnYgn7boC0r0Ga6ul\nSvrAjgVPg6pkj58vQUOt8S2W68APIhiIHvx5EAcrAJk18wkWc+ygF3EsLnLuSey9qwNudC0+hJU5\nEp/Ddf0tEvqeDFwHiztYHBpXgzg0csv3VERV43EzevDnQRysAEg+S8JgvldbbEgSmBKka+mWzHyh\nFWJyNIJOC4hs4VJLPotttJDPzZbZy0cj7V2zqFJXuZh7lmp7zqAPSMYpV4g4HXGwAiCBdIt32tjW\nENlk7d/FE87f2mOHnSxqGtCQA1rtPqi8KxBOmjD0wPOQvTy45fm2JA1ASY3eeHyMOB1xsAKQiY/n\n08q68BteX4vQss9DQm3lfxEb4lErGhVb62mSXLXVP5ek1h0v2e0pP/HtkuJ2I9w4swIQBysAWcPT\nrvIqAM6I1CcdiMIzuYBHmGR4MEFWl1mQ8pNlUzhv0QolCOzxCotV3fD/OvIwKLNgVbVmd9oLRWtR\nLouD1q6vHGdWAOJgBUBI0VKMcDYknt8olUaNn8axd114ejeunRKC1O5mkrWDeyI6dL7DN82+oYQ1\nTKz8rV8A4Wd7Ik5HHKwAZFq5+/26JegTaEmQIg/jjnjLhEey0dr8hJXBbgjebO3XukkebM3aqkBk\nLQkc3PXKJrxMqd3hunYXkxQ+s/tVER9HHKwAZLLm+5nEVrRUyzGJavdOW+7UripgS4dSPNGIJCZt\nKaZI47CG1OonzC7KpPx9B+isGrwOHRUaViBpTZ5qmqb8FRFnRhysAMTBCoCUSYK24H9b4liWNvHF\nUZXfkrjlhYP5n1mrQ8aBcCUmPOFuUbMclf68q4QOpFYVJ+JorINwgLAdlXiOQtW6THsZ7RhInwdx\nsAJgbbJaRbzm38r2tgE/Nhan2fWMMZNB0IP9D9C0vf3OCEXB2StW1ZuYcl47nBxSr/hQTpL15TO4\n82VJroZfoCDuiFecUdvBAF/WnBqLZ3fOgzhYAcjatm1cwYb2JrNVe8R/s+1Dk4QLfp/98BCq7t69\na4TxeGyEPCc+Fg1Zyi++/NIIn9++bQTQcHK2B1U4QqekNApW3k1tV6UbFdi5A1hPmGZvszXiDIiD\nFYAsscp86N/W3BPguay1xmRmTR5/Gy6xXbh0ERdXlz+hFzEdtt+/N0JeEw0zVvr4+wdGuH79Bt86\n8Qb+CPSKbTrTFoF3guJcvlLZpcbMOkmXt5Q2RpyGOFgBaNndCd/fYQ1ylo6Jyf8X+TGZuV5X3njz\nxroRpqbIBf3mm3tG6A5njXBwdER9YtZfnL3g99M6g4cKRWTZvEIoT0r8PLhStXcsPs6sAMTBCoCY\nlMqLqqQw1vb6pJCBnTdVOQ+CAjjR8fbtKyN8d/9b6Dw+PjbC5i+/GCHNiKTXrpOw9XLLCF999Sfu\nFPWqKqQeIvUOi9f8OR22ffiZCvldB8mQW7UOqPzDOHBqO86sAMTBCkBWeT+XIlV6lt2QX3GQ/U9q\nX1aF00YOjLEvOneZrJvqiDVMFQV3U3Nz1GyOXNa8yo2w9YpoOL+wyMq5JMi22rUwivopd9wtnFq5\nYeOJPSfv5EyTRGsYjjhYAcgQOllzklDVYiPQLFMwgkix8hlLsaL0f3BhetoIPzx5YoT5K8vQeXBw\nYISpGaLh/v6+EV5vEfuevPjJCH/7+z+M8Jc7fzVCryuZUuvnlOhKXoBE2hFg2cUVtew+fNESzWKt\nwzkQBysAcbACkB0XpXNJ9kUsM4/cccXubJmT/52mXW5BQ//zTz8b4e3bX42wf3hohPxEJRScD96w\n6Q2MsLh01QhXr103wmBIy193YpJ7YvWZ/Ymyoe6N+St6aYe/y1udJeQQVVhwk9oNSOLMCkAcrABk\n9+7/10jwtuEldKzcU6/DfnNN/vrkgPzvJCEaNglduXdvwwgbG/eNsLu3Z4SF1TXoXF4mN+Lp06dG\nmGNXfmVlxQjrN24aYW2Nkl9vft02wrgQHoJZ45w2ipBTyziQxg6TtfdLRCtKey1q4SZpcC9EfBxx\nsAKQvf+wa6TBgCxRxkmlzLKGmoPJNSbIzDTlgvsDqkJ49uJ/dGuGMr/r69eMsDMi13x6fhE6//Xv\n/xhhc3PTCCWnqO7c+bMRZmcptH786LER3rwmGua2OWQTdshmt9MhIwinPpX9Hg6k4dNbNMTeKtYl\nv4Y64nTEwQpABpNSHNAEnp2l3FOv30W7hUt0scPcHI12jbC3T/Gw4jNqv7tJlmtpiUi3u0c03DnM\nofOPf/i9Eb74/DNqtks6+/zqmRnyRY8OaJvnYH/EfWeiWdVRiIgrzohhdwe0bbyAv2yj4W9UL0Wc\njjhYAcgSnszb22Rl9njCPzvaQbseVwpcmiVepFLaQCPe53I9mNGq5NxQ2bJBsrJ8hVRxVT4MMRzj\nfEz28ZPFy0bY3KRUV29yILqYUKMRkTTPmYZcnIsMV8qVvzCCRdFCQ+tcbsxnhSMOVgCyhmfdxUs0\nz1EOW42lWLbhY9mDASVzUQePCp5KUZuDQ7KPBVfyjXMOPGsxYTnzGDSE3cmYKSknWLocga6vXnUe\nV0qV7HlWnDhqeM8JDNOpe1K8kjNDkjgqeenAmlDHFM05EAcrABkog1mHdAccQqWULjkvyns5OVfN\n9jPKzHSEO8je8OOY+aX1Yww1NjvlPdyM+ctv2d+jDmRMzP60dC/nOG5+boaUF2TT9yoUPXT4HbKB\nRVcSoXQxphdVXAQMWxlnVgDiYAUgO2YaznEyBDwBv5RSyyuU1ex1aTI/evS9EV5uvTHCYEhbCUh4\ndlLyG3WXnUxl5yS50LxyDWuGA6mcGtIDEsbwNot9UcQBYMo1VDOTE0Y4PqRDL3VO2VosF3ND3h9Z\nmIcq1Dq8eU0PVtXgRHcjzoI4WAGIgxWAbOEy0fWIyzQS9iFu3/4M7VaWKTO1NyLmT0xQNvnwmIz0\n0xfPjfDkx2eknVUhRzbJJ+GU5a9P8PrS4aieM2MSig/6tHCguPKoOIYq/KbTaIeC//l5itKHvJIO\np+gtV68sGGHpCn17t2M5NLwX++7dB/5k+sA4swIQBysAGfI+MMljrtPf2JDK4offkYBULJJWq2tr\nRrh165YRUGb14AEduHn+nBi6s7MLnb0eu/68EwNh0KFb3Q7Fz91u12lTWbWNSUqdQeHFCgf8K4ur\nRri6St7PBU6E9bFzbKnCNm2vR+m50ZAS7nFmBSAOVgAyJGum+QDN+JBouPVqE+0O93aNAIp1mBf/\n/PprI3Q9WoE7S0tLRsjzH6ETaazhkExkxldqjl1hm0bcAcTkCJ6VUkfHtIZ8yiVKO2wWYaw7XVI+\n9SkRM0mQ/hYavt+mF/X7ZD3n5siUx5kVgDhYAfg/pQ4eZ65sAxcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<PIL.Image.Image image mode=RGB size=100x100 at 0x7FC125FC22E8>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(data, label) = trainset[100]\n",
    "print(classes[label])\n",
    "\n",
    "# (data + 1) / 2是为了还原被归一化的数据\n",
    "show((data + 1) / 2).resize((100, 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataloader是一个可迭代的对象，它将dataset返回的每一条数据拼接成一个batch，并提供多线程加速优化和数据打乱等操作。当程序对dataset的所有数据遍历完一遍之后，相应的对Dataloader也完成了一次迭代。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        cat         dog         cat         cat\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAABkCAIAAAAnqfEgAAA46ElEQVR4nO19SZNk13Xel/OclTVk\nzd1V3dVzN9ANgMRAAgQpSKQ5KURNXnjSwhH2wuHwSlsvrI0j/AMcdkghb62wBsoiLVIkQdAEmiCG\nBtAAep6ruubKeX6ZXtzvO1VdnbVXhe9ZAKey33Dfffe9d74zfAfw4sWLFy9evHjx4sWLFy9evHjx\n4sWLFy9evHjx4sXL/18SevqnC8+edko2leJG0TiAaDzq/oxvbTolluQGkSPHnZIc9JySn+NB1q+9\n75TwzqpTjk2PO2WrXgfQjnCXXkCl2ew4Jej3NUqeOtzlLx2EnbK8ueOU6RwH809fWADQz424PzMn\njjml3aw5pVRp8ERdXnIsynnotAOnzE7POmVlq+yU//Rf/gf2yL//fR7/2QvcslLn4SrNtFP6sQmn\nlBsc7eUPb3NaKjzRpRdfB/DROz9zfz53dtIpX/mNFzi2QdUp8Q6VZJxHq+2sO2V0wBmrbqw5JZFP\nA5i79Nvuz0b8klOuv/f3HNLjyzx+rOKUVIITuL7O4+eKvK2zp3gh07MJAIlQxv25ttrk2GIRp/zB\nv/0zPCl//G/+HWejz0ve3Nzcu8FIhjO59ogrZG2D19Vot50SYMDRJuI8WsCjDQb8p1w2x72aTQBJ\nbdkNc82s7XACo+EkT13gsKN9LrBGmdce9HlpOa2reDwBoNVqaQMOAAEHkEoknHLkyBGnHF/SBM7O\nOSWZ5Kn/1X/4o72T8Mf/8U+cEuLBEItyJJFwWFfa1y+RfdfeH2g29MhEI1EASQ3JnvNwhPt2u1yu\n586cdMqpU3xS/uRP/rNTPv71u04pTnNZvvTSywDe++kveZAQZ+mNr37JKb85/6pTOjVO1PYOn9BS\nhY9Su6UnpVxyyrVHtwF88OhzDj7De/fe3Tt4UsLw4sWLl0Mi0ad/Cgd8SYd7/L6F3Pu5TQsoos9d\nTxu0SxvcWR+Iausjp/TL/Fpm9XLsdfk1CDpdAEjqK9Hh7/0eTxSO8fsw0D+hWecYYvxSJaIyvvSR\n6XY6ew9y+7NPuW95m7uk+UkfnTvqlLU1ftu3KjQ3MrJiIoPh7/RElufdaXNI0fSUU5Yf0N55+6Mr\nTtmscqIWjp9yyhdfOuuUN37rGwBK67SMfnXlA6c0evwKfe3lJaecmqFdU2/QVEzkafhEwY1rG1v8\np1QSQDxGo2NxiV/RxsbPnbJ6lx/AcIj7Lj0juyPDmezIuGtXeEXNwjSA4pEF92dxgl/C2JB1RMnm\nOAazC6JR7lWrVQGEYzH+qUveqPEumGGlJYhBwJFkM5yNjqywVqXEK4qEAVTKvC+tTltj4Xknpjik\nqSLHXa9w4+oW57Yvm6QtS0TWvv5BSlimUDTKC6k3aXesb9CWTKU52rFCAcNkbqLIo8meSqdl2Wly\nBzYbMpi6mrFOh4oZgG6LVDKx/0wafqfNmUwkeN/HxkapjBMD9cBb9uKlS9ytvgUAUU7X5FFajsUM\nd0lFOMlBlA+gGZ6NiMzYLncv12hzrZTXAPTAXRZPnHCKt7C8ePFyiMW/sLx48XJoZIgpH5HlBmHD\nkckpACG5OXtNOq3jyaz20S6tmvalddeXRR3IGO21uXui3wXQE9zo6XTjYzxso8Nf2gH3NTQRjlGJ\nx+Wq71Jxx+tuEqXWW0QEk3GavjEeFZ01wtXxDM3vVp828P37D53S1aj2STxFONno87C9JgHarfuP\nnLK6zaP15OUtTNCE7g/4T+tr6wBGRokItmuc5Hfevc6xZflRefY43fC9PqFBRG7yuPzKE0c5CTul\nhwAeLV/TnPCwN279wikhzWRboYeIEFOhwIMsrxBg1hscbXpkBMDDCH8PdCvR7+AASaV5N/vyB2dy\nvJuxRBJAU5iuEXAk23VC0YHwSz9sQIz/b3S5VybFEId5wQdBH0Bbh+0J+4znOZILJ+gUXzqed8qj\n5cdOWVvhtPd0gwxRxpMJAHG5KTpa2CH5wgP5Q+yfzM08eHjfKd12C8PkzGli9kF/vx8d8qwHu3EG\n7abZMOd6QqGGdCqNPXDVYhR9jdY8M3Ht0mjxvv/e737PKefPPeOUbIrbfPSz/wPgxPy0+3NsQeGF\n/BGNNrTvjIZbA52xrSl9LH9RuVUFUJyjU6W4MIUDxFtYXrx4OTTiX1hevHg5NDIEEsYLDBZE+zQR\n586/DCCscNjDHYa0Zk9eckrxOLOuSiv06seUXPPgI6ZsoKzsLYGRQa8PoC8oZ8ZqSCZvRoZoNKxo\nSJMGZ0ghp5iihF0FShqtNoCkdsnLfu4I3A1CPKxFXbp1ItnFBYYnNjcZUlxde4xhEktyy5PnX3HK\nW+8wMLrdJCQcGeFMrik96trVG04Ja9jXrt0A0FL007KHLBHswQph1wefrjjlyy9ddEqtymu8f4uH\n7VvIrNkDUC8TidRbJV6pImZxxY/yUeLKSolD6kc5t8ksMVR1m9Bma6MJIJFlOKwPQqoBhF+eklqD\nIbOmYmcWyXIg0aYikjCFxzc4ZijYMo+CHs8YUWJRLK511Q8BCCncFg5zl2SEuxTz3PLlS2ec8nma\ns/H++/c4WmXV2fFDkSj2JILVW7wvvY7CbXJThM1DUudybXW5TaNaxTCplJmsFNdVxDUJCaVuJeUB\nqOogFrG0GGZVYe52swmgF+zHlRGFMsN6gmpVxexK3Ncyv06fIlBtC1a/8PXvAHi8fM/92S3XNP66\nxq/D1vlL0zLXBHK3any4Vkp8LjLjaQBLzzOGnh55Krgp8RaWFy9eDo34F5YXL14OjQxLHFVEJq4c\nsHg6CyAcUkKpLNHJo8xpXLhIWLSVJxKcmKcxWV6555R6hRGBdIpGaSgeA5DIF9yfgwoN3R2lkyGk\n2JPS+EIKJEFKVOZrRwZnP2gBiIZ5Fsupi8eENEOWmMoA00pdJSYRnmhinGCnWq9gmJx79nnuUmD5\nxQef/zUvuc001GKB2XTdZla/8GKhYFOz0wFwZJalD40yg5uRCDeoKoq3ss3xb5d4pTuPVZpTmNaJ\ndMtCTQDtesn9ufWY2KpWUc7knAKvHYHfDSKO6VOMDVU3eCPCWifbGw0Ap59lEGd8ZtEpb/7s5zhA\nqprbUomDabX4i0uS7CrF15BOWiVfhr8GWnIWgOsLpPSFDeMpIcp4BEBbIbRUjjM5NcLDHp1WPu0M\nMfv6GuOemSSXxLYcICEBZFeatlPlnDQV78totAbZLLhZFxwOhI86ieFgp62j2eSEIlzYFpfMCqH3\nhJTzetwMGybiXPaRaARApy1gq4PEIvttFEPZlrO6mxOrX0ZGCk5xNTpj4wpqf3zLKT//0U+4pYLC\nc2Nck70W79TKJp+L649uOqXU57QcXZgHEIR4XY/uP1G8tVe8heXFi5dDI0MsrGiPb8Socvbb1W0A\noSa/LTHVCoTktY6q+FY1sEgmVdoimyUqn3dCBdINhAHsyI7rp6kkOvzKJVL8LLRUSBlUuW9CldgN\nS1fRYCKROICYDhviJwd9vemru85FugxjaZ5xIMOtLidooJjAPonGeTmf3Wau0/W7/Nrk4jzRSJaH\n7Y1yNs6fveCUZo+j3SpXAJw9Oe/+XFumw97c8Osb/C5F9KlfnGcKGBol9//T8+d46iTzvG59ug5g\nZ4MbrGzwUzxZZIHr8ZM0Ia98+GunLJ3+tlOmTp53yt3r/9Mp6ZBuRC8AUJaVkSlwAtdW1nCA7Dpf\nm7yJnSentFbl7ejsOuP3u4otKSmpOp5kgnMbliVlCVnOZ5yUL/+ZCywkeu4UlVe/8KxTsnLzD2SG\nh2Dr1nz5PKOrcW7JFEpneDorGg+HnjJelHrWVj2/IYN9ktYKjOkCIaATCat+qEYLtKVEyJDO2Ota\naQ5n+/HyIwCNBrc8eYr+7MlprhBXy409ddGWKGc1QHYfLKnKzU9nlQvg8ytXnXLr4V1uqmS0+Ctc\nYD29HG5vcW1vBBxkplhwSrnRAFC5b8a48vueEm9hefHi5dCIf2F58eLl0MgQSNiRwZmT7628fBvA\n5ARdlRMzM05ZvX3FKRHR4vQEZIwDoFstOcW8fQnhxlQ4BiA9Pub+VLE9ahXu21blR9eIIhKsekeE\nlnNMR4tEVZ0QCgPICZyG0ryKHRVFVPscSUZW90AW9eYqc53qHav8GO4lff99Xvu7d2moB1Y/FCk4\nZXWD0zI6wRl79Wtf5xgaHE291QGQV2bT2++QpqoioBSOEyx8dpsZYY/WOclfOHVUo6Uxn8trMptx\nAJOiozhxgZgxE1fmVIxW/Rde+6ZTls79rlN6ilcUF95zyvYdFqxEonEAGyvkLerVOaTTCwUcIPWy\nylOUZ5cQEHPYsB8W8IgptCJ6skFvf45eICgdF1nb2BgdwBBkbjYr2DOll84QCb72PKHu/BSLqNpt\nBhwiWkXmhYjIXz4imq1QKAQgn6NTOSWHyS7/gWpQ4srDsqy6XdjVaWKYlJU/ZVA0oqqarAYQ0RM0\npkdmdJQX8vAB8x/f+tmPnXLlgytu0O7PpQX6HJ554WWnPHvpC3uvC4AhWqv8scFEha/R7QO4+bnc\nIPfJ79ZXsltYvprkMd4X3TpMJYgN4w2GmIIQb/3G5hqA0jY9DJvizHhavIXlxYuXQyP+heXFi5dD\nI8PysGI0tkenaUaOn7oEIFxl1k8/ygja6MkXucHRRaeUHtJEDAaKMKooIazwQa1FO7DW7wJIpGgM\nG2frQJyzQUeBP71XW00itVhII5fVDcsiCXoA+n1uOUiKtu00OfNS4lleqxBx1EXVunGdwY62YppZ\nlfjvkw8/pFV8f4PDTsQIVxsdGtL5Aq3i136TsCs7zgymUxdZ3e6u9e1fvqXfabF/8vGHTumUaB4P\nBJSufsZw5KWlLzvlzHFmTl3/lCQTocgRAIkE81mycV5Xo6a4lSz14tHnnJJJEmkGIcLG4nHm2W0r\n0hetPgAQqj9wf1abVArjx3GAxHXzQioK6SrklEzGATTa/L2nUGDPQmlaM4atLBRYiHO2jx7lKjVG\nh48+fABg9ijhkiVbjSSMlJEnauq6AkWycsrVyqnIKbpLVRwBkIgz3tpQftmmas4iip5HLBFLK3k0\nxzHkVPe2T9Ji+NsTodMgFV2ty2mSFf13OEy4fe3TT5zyyScM2505fxHA1DTX2/VPSAz5s5+QIzum\nKOEXX3zJKcmUQK4wYdju3e7DNQBw5BnGHBNFjmT1IVfCzhYz2mbECl1V4HKyzzsV3mKy4eYOIXml\n2gGwts7LaTSHA2d4C8uLFy+HSPwLy4sXL4dGhhH4KWJijNoD9AH0OsQmtSpt4OPifs6P0/weBDRf\nJ2cJeT6VDd8Wpd9uhXk3ADCmwFZUVHybA4LEYyfJWFDeWeapR8UyrrhYp0br0dIR6/0+gGSSuKae\noZLcKfEggdLtxNfe7vIgi0vkYDh6jBjn8q/exTDJKW8wLRu4KqARqKrmpa981Snf/J3fc0pLbIKF\nCc5Yr9cDMDW36P5MZDkbt+7zkjsBGReMvLBaI37ZVBuYX7z1Di+2Qnj17MXXAIT6BAhBg72LGhXe\nxHKNt77VZWLq3CxDfuubRLuIE5q9/p1vOGXrxq8AbN1iLLVaoUnfqq3jALHs4q4CrzERRTSaDQBV\nxUN328OIm8G433q6uTklWB4/RsRx/ixTYasNQo9HN6IAzp9gcPDkAldRSlHIpiLX22u8kNEs0dC5\nU0TBOw1GQlvKN3auEuOW6LTkrxiIM0++FOt80+4wzF1WWdKgpzD3k1IV1V9MlA+WSmrdqgb6JZ/P\na2y85PfU3mZc3PAvf+kVAFu6QMsBvX+HTpvv/+VfOMUSd0+dpc+kWGQULxRSex7llDqQODHLspvx\nGULOaUUhV5e5bhsisSjt8NKMzrDZbuqwnKlSuYY9tVnBAem18BaWFy9eDpH4F5YXL14OjQyLEqqA\nK52i+eqIojfVP7WyRcsto36HyaiYElQxn1JsDjImI9axSceP9gcAGptKLFQm3rao1mdG+MsXz9Ds\nH5mgsfreR7Rstz9kF69ASYbJ8SKAE6e5y06UBxmoudCUyN1bslofCqdmU7SBxzL85eTJJU3ME9jw\n/EmOpLQhkry6yveVWPuFlxlFnZqd0bVzNqx1q5vtV77EPpSXf83gYEgVZLt9n5RvmdAgt8qs4Lt5\njQR+J44xbrh07hKAdpN38P4nhAa1Bkd792aJR1tlhVcixmaufUHy8WMc9swE4VWutwig9JiIIDkg\nuEvnp3GAJIVoArXP7Ij6zvXRbKhEzrgZCiOEPFYKZ0VzqkxFW5SEqQSntK+Q3OL8BIDXX2b0M6vg\nYFft0Vp1depVGufkOBHNBBE5ZtU61NruVkoN7K1zVJ7weJ6Okbhu0C4jgnI+00LBQX842LGKRQvR\nVdXrzHBxXwH3qBZwaYePZE15p4vHuFw//eAygLVHvN1bqkiNKpa6ssyA8l/+BStGj6nt6ytf5iq6\n9DyTS42Y3w3PCB56osqwhNtJgcRSibkEUbmYrAbZrCRr+OAyBGxu7bBPi7ewvHjxcmhkmIUl52jX\nfGPNCoBAZTchUSdXt+mmbTdLVOTODGS8tBp896fFNJTK8G09iHUAxOVKjIgctiEygI8+ofVkBEAX\nz/NF/ux5ZoLce0jbYXmV35BkMgUgHldCjZJWtjb4qQzqPH5FaThbSgwpWZsW+9A1h3fNSYZpBr76\nAj8pLyaohKaZFXViSQ3KE6LZlb2pH4AgANC2FB7ZU+0ax9ZVC5kjC6QYe+MNGm7ZOC2sasUq43nL\nrnz+KYBnL7I0p3iM7F2rKnqIyCjeXKWX9OGdklPOn+OFNMRl9uE2rbCp2ecBzJ15zf15/SpZoZc3\nD/wkGoOwWdbm5R30AgARlYZ0lfqUU+9Vy7oyd6wxOoQjnMmNTUUAqry0+dkigLkpGktdGXQtHSSu\n3kjGRyzn764beFLe66YIr4NeC0BEnvt8TlzDsv4MmmRlbsxOM7RyZI4W6LYyj/AP2CvW4yedlvW0\nzS2vfnzFKVubtKfGFeMyc8keydoOV3JzJwDQlPVaVzKXJRh2hEja24wadfSAN0s89Y2rTO+aPUq8\ncvH55wEcXVjkuHXvYqpGCmuFGxl0qL8/6lJR+GtjfWPvPxldRN/ux1PiLSwvXrwcGvEvLC9evBwa\nGQIJu2qnYYkthU4LwMASqQRnrJhme5twLCzHp/VY7SjLPidPpJH8xdMZAJEMHcPViqEVuZnV8nRd\nzTlu3Rc9wBkivpNLi0756CoZBTq9MIC2XLyP68Ru03IHdpsqdtGVToxz45ici7NHCaZu31Mmy5OS\nyHIAC3NU0kWmnrXydFGPj7IOw3ztUXWXCVu/yRAAREU+8fAea2ju3CTaChQrWF3jJMdTBIDjY2o9\nm+NhV1eUp9beAHDvIbHJ6n2Cx826PPd5Qp6lokqXxnmn7jygOzaaII6YnuE9+vTquwDGJtkk6egS\n+Qh31rdwgDTl6rZ6FZuNQb8DIK7MJuNmiOu+hDQtvQRv0Kxu4jPPseGNMR0OROGdH8lhTx+alFZR\nUlQfYdVmxcL6JatkvQxPVFau2bY68rrWTYVRwtW4mPYEdBAoN7A4zlV0TpGfuVmCuFIph2HSk0P9\ng/eIwj587919A7BGqsUCvRwjqhtL6jluqnzHMVwae6IVu3TUUtfIES0qEtEEbqpTlClXPvhAw3sX\nwIVnyYC4dIp3YXSUkYeebmJTo20o99ByAEOKcQ2sNWy/hz1k2eGIeUz2i7ewvHjxcmjEv7C8ePFy\naGRYI1VFrKxwvBf0AAQ98R/Ih9+xdAyZdtl0wSm5caaxGBM5ZBmG2uolGc8B2FY0wTjMLG6IECGJ\npXLEFTYqlxjpGxU0S6vf5E61CaArpNCDunIqaaW+wxNVxeRtletR2aIbihsmxcu+T5LKvlndJHZr\nbzIadeG1rzilMEIgYCRmEU1pTCdq9QBAtHTYUWyoLg6AkEjOynVO4PqOAKYYKU7Miu79CDOntqpN\nAD/62791f9aqzMfJq8HtkSIjmCfm+Eu1+plTrtzg3M4tMNA2r4y8x5/dBxAd8NrHThI4Z6aGzxKA\nQJ6EQFk2IXX97Pe62BM9tEakVggSVzZfWjdoYoxDSolI7/7Kg33bODaIW3fuuT/HR7lCJke5JtPq\npBsRMXw0TaTW63P+t1RxlUpy/cwdmQMwM8N7GksQ19y+TvDYqBDWZdTQYKrIw84bY0R2yBMH4Bdv\nse3QO7/8v07JylWytMTosPVYuHuTaXdrm7ytcaX1WcjVwa6aHrqaSOX7QuZJy4pSqDGqg+QznMlO\nV8QqCjLevXkNwIN7zIKcKBKhX3yBkeuTp+guSKvaKaUF/PmVK06ZPrboFCsGanS7AALZT0b635RX\nysRbWF68eDk04l9YXrx4OTQylMBPVQLWvbLdBNBWnURXBQ0rouuz5kUD2ZDlHVrU1udyN51fYYpo\nIoY9qXT3HzBDzyryM6M0OGPifkuklFwqyu2QeqxmszShtzbXAZTnCu7PqWmGwwKFMttNy3K0fE6N\nTS/wuvgbsoXh5fXZuAoOckRh6QSDg8ks6QHqOlEuo8Zoin8pmopuLwDQaIqNIE/skMkRetQrnMmp\nGR5/bHLRKW+++Uun3C8SP776BdrY1WoHQFBjcLCQIIDKiSiur2ZumxYFUyrp1ChB4qkF1nkkwjTv\n8+lxAEnREnz8HhnoA5GjPy1NhZKNtqGnWpOg28WefNGQMF0kZAoPklRpS1lUi21Fu0YFEscLjPRt\nLa8AuLfCOcnnxJWoIFQkQ6QWS/NCWoobbm4JktcY0rr0RRJ4pNIxABHhpiOLvO+VbQ6ppXDbaFZ0\nIHU1BG1wAg9AhKgrPfjYghjov/obTpk/Qtz9UMU0j1cIQrOibXh4l8Fla+/q+CwtFTMlAJjRIK21\nQlSPeVaTbImvOwEf+ZgC971OAKCurOZ2W6U/m4zFX37rJ5oEXvIJuSkENHHncyaEH79w0SnT07MA\nAiiovcEI79bGYzwp3sLy4sXLoZEhL/ygzI9Mp8v3607QAtDWN6crn5i1oTdXX6AUp0D71lWsM6JT\n9VQ0E05lAVQe8StULvMjM6bKjAmVR1gn7qZVZsimMwsum+Fho+EegEGYtpjRH6fkXEzJj26GlhmP\nHTP/ZGbGZE3sk/FRfty6St1qRcUJK76qpIpFzK8ZSVKJq56j1Q0A9OQKzY7o2qdYjtvVx//ZS190\nyrFTzIL5/vdJaZQI89O0VRE7cyoGIJ8W6VXTWrAUnDJVZLFIXuPvdLjNS8d5ITOzdK6XVKtx9OgY\ngPkZGjW1Mr2/q1s7OEA21TUnp++25WGF0QfQ162MKjxi5S99VZwkVBddVdnWqkqyFo7xQtJxjv9B\npQxgapY5UMWiTGAZxWE1WDJW5UbL7E3jO+YgjyrysF3aAhAO0USNymaZ0yxNn+ZyHbGlqIBDQzEc\n4wLbJ7/1DZJot7QU4woIbO1wblceMLzwdW3c1uP2F/fvOSWlSU7EYgDyGkmxwHVlTVjLNSojilbF\nZdBajyKLv/UUFYqEQ9iTuiXAgGxMtOYtDqnS4/EjR4mT5qc4UR0Fu2rbRADZsVkAR44W3J8bpas4\nQLyF5cWLl0Mj/oXlxYuXQyNDIOH2Y9ppxpUcDS8DyCh1yDzHazfZvsV6ZBrplZURdOX2O3GabDsL\nzxLaZIqTAOoRWqRpJbyMqfIgIRs+L4d6Sk73elW15vLphhslKpEegEiLhnRHmRwP1ebkkZBaO0GD\nOSN/7biYi8eLNO8nZ+YxTJLK3Gl3OJJ0WqTMTdr/6+skQugFtIqjY8RfETEsO96rnJi/ipawox6Z\n20JbS6fo/R1XK5TmgEfrJzlj40fZdOfhjSsAQipGiYR5uu02zf5zx+nvXDjCbe7eYSpQtUnYhZLg\ngxzzrUYNQK3C79z0FCchHFvQxHyGJ6WpVKCElYAo/OJad1ppvnX07Aky2C9RRV0iWnurqh+aUECj\nHvD+TudzAF66ROA8PqGGNFqK6Ku/qZZ/Te6IspgkrH9ur81fjs5MAchliECbDc7SRKHglIsXWKiU\nFrlVoIKY9UcMKNVUqLRPNjcZWjEusOKkur22eJDxUd7clNbeT3/8Q6d0FAebKHIvB++SyoIMW3wp\npN5F4geP57g2EuL4NrbxsPlizKcRCdt/AfQHqqbSDeoZP7T1l9UTGrUePDpRtE5XQ3x0HEBhinNr\n4PRp8RaWFy9eDo34F5YXL14OjQwrzYHhO4W0EAPQskanYEglpj4uu8a8lHSu4JTXXmKbxmyaRuPs\nOXLXHltaBFBt0uSO9hkB3FxnTkdHCDOcVhpIV2Rg1mZSiKwlJRXvAth+tKKx0Ri22p2kwpQTRwmy\nJmdYtJFUVCWuMoh2a39xAE8nhtxcgfs2uhx/q8H0tEZMHHJ9ZcEoxS2RIORwXVgiCkXOzDOSoiAn\nQjHVeczyRDHV5sc1yR9cYxrOwhJt7EQ/C2CjTgCyrTBfbZXQ49rWm055/XWyM59aIi3fOz9kQc+D\n62zG8+J5AuTFiRSAboNHS2Y42udOfVkT80M8KXX1vrVQqeX7sMbLAqYKAUfkfMikDTgLG+oTWxxR\njZEoMY4fIXPAkekpAGHhvXiekLCjKORA8dC4CorSQsqpFO+E+RweP+TcujhYTG1vWsqcOn+OXJIj\nciwYl3FcbBBpjXZV4HqfxJXotKO0RAuIx3Tt45O8C3GBa+N4yI8UnPKt3/lDp9y4cR3A8m32/lne\n4DNld8EisGsbJaccFZyMyRVTKfEam21jYQSAqDCvsB2qqt3p9fj2SMt7Y71jLdFyRbVxo5qW8v0H\nAOIKr2fF8fm0eAvLixcvh0b8C8uLFy+HRoZWCtCoM56tPnoAJifpwy/KmW/8DaUt1fcfZRnE6YuM\n0RyZZ51KW9Uox8+ed8r4aA7AK6++7v6MqaFIVtitqqqUvoioA5mmfYOE+uX6Ju3MwswogLPTLGio\nKQjlOLkB5KNEAqOTDLcVZ5gqudvsM2GJo8JmT0oblrDK0a7fu+eUjoztcEa01hqtIU2jKHMFK0Zh\nbURrG2ILOHaKiOO8WNNS4k74J0ogvHObCZwjk5ztUHcSQK/EqwiJ6/22CMJXfsXRfn6DynPPsaxn\na4Uo+MN3CKtv3WBI7ptfOQ/gpUuLvHbN0uj4HA6Qnua/qihtTIx9jqchUNDQ6NxMokrfTYh8Pati\nmpPHGJd85dIlp1i4sLKzAaAuLvnUONkOUglCNsQV84pxm4mxglNG8gzAdVr8ZUm9dROROIC4vvFn\nLvB2TEwS19TV58YCZV0tzqbibil1GNgnlqI5M8uZDIu80Fj01hS+b6kg5t69O045okZNJ07yYp//\nwgsAVpeJwj5491dOGWjdGm3D7etcPDXV1VmK70B0KdbsJxoaAGgrjGsrN6T1PAjv5wXcFAC8+YC1\ncak072ZbT0q9UQGQUqgxHDuQ/MNbWF68eDk04l9YXrx4OTQyBBLGxG9tbHaOS+z3/vk/c38eO0Ej\n2XLP1h4TO8wo0JATr97GI4bMzpxTZba6inbqFQBhRfEmBNA6ai8WjqrlqnWmDNNorKoSKhDSTCub\nbi05AqA9z7PsPGZcpq2cwMQoDc6W0u0qqnczCQmM5JQOuk/iCoI8WqXF3tOwx4vMNQ2E3aw6MqoC\nMQgWudhop8NJuH2DuaatLi32F19h8C6u7NCYKge/951v89TBV51SGGF4ZWV5A0AnwjBfYZyRmtU1\nBqHW7v7CKY9vkUT8wQ0qFinr9DkJV++VuFf5IwBbXV77N18jAEmoxdnTEhWa64mVweKGLjQWCDMm\nRNdndOkNUZLnREswPU13xInjLBWcUJVcRL3m+q0KgLDWTBDwdBlFJ6MKvPbk00iIznw0ywkslThj\nZ88xz3lhdgZAuCfyP3DfZkddy3b5SKQYl78RkAyGp0RuKT243uzqkkUer5hdUQ/XzZtsX7Atbonj\nL7Dj6ZaikKWdbezpY/al17/mlFSKTgkr0G28zlDyhlbynes8frl5k8MOWwJqAKBlnBC6mkBFuEk5\nPVKKJH6qLr+rwobTxRFdmqLmgxD2RD8z6eHAGd7C8uLFyyGSIRZWT64vMU1hkEgAiOpN//HHbKc+\nUPrQrFKE6qrV7snD/eHHrNXY3KAXOax8K5ertXqPqSK9rj5Z4kWNyJ6KxPjaNkoja92zWeIZ4yl+\nkQYIAWjInmoqVyUzz2/ylFhuO3YwpQKZl9fc/NYFZJ+sP1QNR5rX3upwJKM5er6nT7FWozPgF6Ok\nJulhcT8lIxEALXkxtzbIFvDSy8xfe/U1WlhWD1Gr8Wu89pjm2CAkfgh5wbe3agA++JCuVsu+2dlW\nkU3behSZx5/H394Wv4KKNmJx2h3b5TaAy+/zvIvz9NMvnTmA5wnI5mgctSOixGib9z0JIGxfbzWR\nNVOlKoKQUXmvo7JMk2lOYEPFLllZUqlEBEBLfCG2FK3bqNF0hAJlDykZcGmRsZrWgNZxD6ISG5sC\n0K8r70+MbLGYjV/0xLLX+oIgUXvQ2sP78k4UmdGWUMSpo9X5WPDF+v5aL5ylRS7pC88wAjA9/YSp\nazNpRTY2ySZxLUULPY1PEZ0cOUG+49UVOu9LmxsAauLMsNS5juqH4oJl+SwP2xRzWVge+q5ytaZG\nmaiYzIwA0DOHWGJ4pAvewvLixcshEv/C8uLFy6GRIZb8b36b2T1mRrpWlzJvsb5O59zGJn1+d27S\njl1TQ80pOQjvP2LyyEdXCf1qFaKepZMnASw/YC5JQ6ZjTU7Z7W1in74xC/f2V+SEk0SCp87Qjj1/\nchHACxdoLffkGEyrz00ytd/qzijOEDLvvuBwQowO++TIDK3l7CjdwNsZ2bcFmtbRpOiVQzzIwzXO\nT0MYZ3Yijz35X9aj5bvf4l2wTrEfvM9mln/3v//aKZ9dZcOeV19jZcyE2CZ+9Pc/BlATlW1XrMT9\njspf5O/sCiilxZ589jxrXB7eJ2PchtylmewIgFMX2BZo5tRXnbKsjkRPSyZJXNmTp7ahGq94LwLs\nen+NBaSlyTFuBqtBaaogprTFpJ4HqqqZzRODxKN9AL2Gcoh0l3s6bKCj9QQJGxrb8TNcNukZpr8V\nirytkeQogKBjPZ8IFQfi1e7Ip9FQE9+tbc5bWZ2Au8MdDFjfIFQfEVVJVjzOeQUcEimecWZOjNWq\nzbIHxAgIXe1XWvGThJSk0fVZMMS4GUTtYNwYR+YIkBtNQk4XnqpWDBLykneUNmj9a8uiKklUuQhH\n1fY1ECS0hDVXixMWkXdYvqCnxVtYXrx4OTTiX1hevHg5NDIEEn7rW991ioU/EvEEgE5LJNDjrBq3\n1jXNBi3Dx+s72kWtPU8zUmY8bSPqQ9MJxwHERpm1lBJvdKygqpdJGuptlVlEVCgTUrAjo8LuERWs\nF0ZGAMSU09FXoMQ4BQcCtxGZvjVFoyx5xzJo+v0KhslOhdAgrEStlga5tULzPj2t1rM6UUnNhIzD\nrJ1JAsipxcjLX3zBKcUig4/W0fNv/up/OeWnP/k/vKI274i1WvmtN77ulHffeQfA7etsTxJWGHRc\nnS9HVcTTUMTn+HH2yPmjf/mvnXJPkPCHP/iRU65dvw7gxj3CsT/Q0ULR+zhAAgFAq1Ox1qqOkMAQ\nuiXZGae4IRqr2kmohGVECVOtDpdcRZlr4+ksgHiMZ4kNeHw19IQFyrpdgSC5CybmOAnTIUKzlIqB\nIvEogJhoNDrGUFLnkGod3qnNHZWsleXc2FKbmc5wTGjLWJeOHflDurrLCaX15cU/kc9RSVgTWc22\ni6dXyly9DTkH0soIs5ozi4NH9cjEFCu3AG5UdUJT09MAplUJFFHQ1ig8O3Ls3L5+zSmX3xTLoEp/\n+nK4WF/h8sYqgLFpvgqmF4azZsJbWF68eDlE4l9YXrx4OTQyBBK+/TaLNqyHjwvfWNJmLGzNf1TD\nLSa/MXGWN+q0RbuitesKMT3u006uddIASoI8NVGtW/DA2mdaICOwnNIIx7CtTMjl5SeKHjo6XdA3\npmqOdpc8T9igq7zN3cIJJQH2guE2/LtX33XKkTLh2JYoA8sdVnWcucTg3aaYJAadhhQBjX4PQDbJ\neTu+SLoLC+L83d/+Dc94+W3uojH0pW6t8fhzk9z95RdfBvDrX/JW9vvivZuksd2P8PiTo4QV/+IP\nyf32/EVyh/e+zBtx9gIrP/77n/0pgJt3WG51V1Tl85nhVOXYU17Tf4qMwaU12q3cxeNt3Q4RH1oU\n1QJMk5MM54UD8RqGVGeTzgCoquNZHIqLySkxEMAJRznt+RxDgSPqCYaw0KiRSfQH2FNkE1WosasU\nZQsOaoy7cb2RMV6IAb190lT3rZwCf6pTQljtRTvC1PYoGctHKi0WEC3ppEv2VsOElIKDKRu/pn3X\n37L7wPP/FmWOqukZeRosDVW3zDIKkike//xF8nTWK7zkt7Z+oI1F0KgIu6s6aqjBYK954HLyFpYX\nL14OjQyxsN5/n1/ywcC81APsebdF9XEO1ALTXvUZVSBbu4243HJm8vSUgR+Lx7DHcDO/nYm5iu27\nYRubO9ActBFlkYTCT/RiGeySTWms+maaFWMFH31dclcnMp/9Ptmp0LLr3JbRJw/uyBQdtyEZbNvr\ntES2VtUXfqBcpGPTUMN67EnY2Vjnlu9eZnlNz76rGtu42IFnVUuRUelpcXQCQEq11maYWLFIcZyG\nye9/77ed8tpXWB9bkas4KVbrN97gP926cwPAxyLV+rP/+udOOTF+YOKMJf4YJy9UueIYdu0GDeyW\nac6zKhofU9taS9FaUXhnakydgcQyHHG8V1HaLBuPyQ6sKUdM1UKVdqDrYv+n2SYvJJUvOKUg4mNH\nxdWVU7mh21E3LmNNfjbPldBURqHFExqiEt4nZn5apVGg1DDrjWqlwuYUt8Vvj1tbe7knZU8im6w/\nnch4qM3pnlM2ohWBJWXmZTUb7km0w7Z1OwbGnafWRA1ZSUeOkyvh7CXa6Q802+NqG3z6zDkAa48Z\nzKkfMEvwFpYXL14OkfgXlhcvXg6NDIGEXbFEGRhydqClPvXFl2q9TMLmhpeLOpOmu9G8ff2B+dRp\nlDpbNCREsOvA02s0FtkPNIxtx6xWgw/drqWTxN1RnLRFw2SK4b5EiMa2WdR2ze22mdDDne4L08d1\nNQI4cebyLD5HX3sr4LQsP6IN/OghuYHOnqTPOJVPAygpke3aTaY+/fynP3bKpx9/zEsWAB9R19Xn\nnydZwsY606BW7pPA6OTCPICpKWbMPRZNUrlEq/vkiUWnvPD8Jac0BQ1urd5zyvQkr2hUpUW3rn4M\noKfyi5tXGDCZuLiIAyQv/GUehl4g53G3iz0RD6MNMA9AaJf/wBQe9sZdXmmpQkg4OU4ll48DSOR4\ntM+ucct2k3eqmySK/9Fl8o7c2+QV5WdY7dTV+nn9dbJlvPHVr2BPx1CjHOgpfzAiV0lGWXUJ1Zo0\nmyotagwHO4bLUnJaDyL7QVZdEaenXSUJoUVzbrhiF5suc7FH5Tl5Gi1aeKpRqesXre2Q+eND2ENY\nZg++EZYFu74UHi00YEDjS19jkuCUKn6ui/fl7Td/CmBknGl9Zy9ewgHiLSwvXrwcGvEvLC9evBwa\nGQIJrfOoGXXO5LW42y5fsSVuPBVJs2hFzIJ3Mk9jqq5At489hqg16bGEKcvXMGgQKPMrohSkmGxd\nI+FzJTiWulVXHpBVhFieV0PJL2Yn93qGVsQ6Ztf4pESTNF9HUtbWVG1TxeT38CHBSK1KRBYJcwwF\nAaVwOA5gSyBrQx1Pr34mwmIiG8wVGVIJVC10dIG/PHxApPnzX3zfKc88cxbA1KyCa4K6SVGjbW2S\nRaOv+f/kKrHnf/vzP3XKqSXC3rQCcD//2T8AECDAoK8us6ESDhArFunvVjvxjO5OBcrLiys/rqes\nq03RGYbVyamtWxYJ8yD1Fieh1iRubXUaAOZVe5RVPPTBOouQ3r911SnvXWdCWVpMcuX77A8UaMl9\n8C5JMk4tHsOe/jpry5xAWx97looatSYt4YtjS1WGZxhZ3VhZK2Gg5TqibqPWOsgy1waCbIbdeop7\n9oMq9rhZLKVxt12TeKKtRieraidra2SVNzGjduj1scf9YpHfnW3VnOmK7O2RlndoXFQi4+Oc7bl5\nYsNrn34E4OZnrCRbPHEcB4i3sLx48XJoxL+wvHjxcmhkCCSMK65nITMXiTND3SBbZLA/068lum5L\nArSDWKsSq39xyZlmzYZ28wn5/6bOaKZv1PizVWsSTlKxqnp32K7CfJYgZ0jEzFaDjRFFW4zT3Uzo\nA/JG0QgxaRO65G6NvIbYLrn/T823dDQepa+KHGj8jWodwNoag3cbm1S6XaKt+Vka0qEWzfLPb7AG\n6PJllgdtbBBI/uAHhIRvX/4JgI1tHiSTY/1QSvb5dXVG+UiRmocPmd36079nef3bcQPvRMqtWhnK\n+AUQKBfz1q27OED6u/fXfALcvd1uAggEKyyn0XaxnFJLmu1r350K92o0t3UqQj/X2TaX5ummjpKP\ncLVCAAhRLR4/x9m4/4jj75RIwpHKEE2nIlwJpfVtACt3GZCti8RuQj1gIlmGKQ0bDkKmaBKiwzNs\nDRLGtcINk1qubF+9nQypGUmDZefu/hPxtRqdymHSU4oyBjysZXT3tXFH8fRkkss1k03rjCkAKTGU\nWIi/o1BvU52A7bCxuMFG3qmonELFKfpVUulXsJclIjLkvcR/OugfvHjx4uUfm/gXlhcvXg6NDGuk\nqqiB2ZMOMXVlp1k/LgvwGWSzekArwSupVjunurCYrGJnBtuW/d1woQYX261Y5/8j+2OOlnHXUasr\ncgDILDc7sytcsxvuVFTHKgcjOrnChrv9vvZJIk8AEh0QdiXUGKqjUOPqMrNAW+2SU+anyKYQFi6+\nc+sGgDs3yXb28A7jfbUSwy49Nezs1nmQbofT0aorQTHKBNGgy0tLxScBxKI6iJIbe4p+WvD08uV3\ndE280qQWRWTAaemK+xzhAHua18bCQiKWWIj9bARR48ZQvmhI3B6IxAEEopnvKjPW0h2bKtOrxSwz\nmacWfyJSqneLRvhTJj0JoDDK35Mi9otrBc4fk/dghQA8tspFks0Q+ywssiD09EmSu68tLwOoV0vu\nzzHVwYX2Oz92J9ncEYaYdkrD+SCtX1ZL2a0WuU6Lt89CdbutuvS4NRoMntpDRE53a+CaEdWfwsQW\nc7QSxZRI/VtK2DbstssLGAwAdASHjWExlycuNhRvjWBbKqi0lrp74sVUsvk8gGe/+PKTlzVEvIXl\nxYuXQyNDLKzAvhR6/7m39W6fSyuv17exr69MLGr1BLYxpbdLh6RteKL9Fefmlbcf7Etl3nGzsMxA\na4rB2R1oj524fwDWFMQ6X9oYBn3zTcok6Q1/2yfFQNTv2i4y5UR6BfC7ND1B3p9ElObG8kP6bter\n2wCaO0zUenyLFtaO2BpC+ma26kzhOX6M5e/f/c7vOuW9X9NxvrzMNKJvf+u3AazJzflXf/emUyqy\nTGwmV1a4izVhjWk28mpzAlEylSsD7CkfSamTblGGycrafgsrpmCI8snQlpUUTfYBdLTMmlpO0V2e\nMutiz4O01R6106GFOCXHbbVFe/Dm/RUADTWzmR6lmRkohhOJc7QFdWM9d4ZFTsVp8l6kZZI8kpnc\nqZcAZOMim84pGSpQ/Y1ayNiaCYFGZUXpV6siJNgn1qXdamisxmWnxClNyhdeUGZWWr7wfagFQCQS\nwx7oUK7QsjM2cEu22jVzcpofoZNEwjr3jGmcSQAbW1ycPe1r3Xo6uonZ3IiuiEezvrwWAdhVEAIw\novoes6yfFm9hefHi5dCIf2F58eLl0MgQSGgUYmZnttth7EmsaLeYamHGnqHFp2GjVZabiWslMg7M\nGbmqVX7vKdYXP8TAAFpv/zZ2fG3jwI6lhhnuM2vZigay8YxOpIsXGu10aetWxRK7TxbV2KMrP3ez\nTjt2fZWMca0awUhVQCxIRHRYArFWqAVgJMWpzli1vZqwzqi0PSKf5dIif/naG687pa6ClZ0abfVs\nPgFgfok0tT9+832dl+5S8+Davn1hqFG5k89fOOeU3AhRTzfoAMipWYsRSGRjBMjvf/QJnpS2/M3m\nc02rRaijLTTyhk6PU91Rfpz5syu6C7YmB8Ls1ap87YKl7oz1lpbrNCMSM2qJCkVFSioo6Qx4X1ZX\nWXDzeI3JbtUyt5keHwEwP6kinhJRat0Wdtjc/5ZsxcPWqtb5Znhf3u0dgvdEnDOZy3DLbEoVOfK3\n2ANoj6o9BcajMFoYBZBRtc0uV6U2MEhozMtxeVHsSS+r6U7QD+mMK9gTWzsm70RLQ+r1nkiHxJ5G\nQRtqFrunhomHdcezaFj3AIoUeAvLixcvh0j8C8uLFy+HRoYS+BlkM7LtAHuiCUbKbqG6Pe89BfjC\n+yN9u/a9JVUNotgTrjKUZ8XodsZd2W3Xsb/Dh2EEd2pLIbEIoBWjB7tZY9ZiRBkoIpmo1Qg0Nta3\n9o8BAFAcZR5Wt8Mtk9OsobGMs16Lv8RChEWZTMEp0Ti36fQCAHHVfxw/etEpaxvEJiMFRltCqrwv\nlUpO2VKfoaWzp3mijEqL8qNQTAfA9777DaesbHLfsmKOKaVdxUWsOD3OYS8sLmi0BAu1RhNAR4l4\nCHjtIwpXPS12W41bLp9XBpCDJ9sKHzfF8GdNjFRH0g0sjKtcLd07Q/EtNa1xtOuGy3bKHKSBoKii\nhOtig6gIzneND1IcEuNjnP+RQgHAmBrc5sYUOMvyVsbV0MAqWkIhnVGc7hbu3Cd55TGFBJeM/d04\nRYwKMT+iihkRBNpe9ri5PDvzRZgHwFpGWfscmxYL/ubV2taIIiJy16yvLmPPelh9zHI046QvTk3q\n+BHNhuLpuon2VO8mZHF3DT90oCHlLSwvXrwcGvEvLC9evBwaGQYJVVhgCaSRZATAQLa3xfUMoBlI\njMjGi8ZoeQaCD7tdmHRSZxBaAMIq2qO7h7c01P20EHvY+Ph/S+d3iZaBhtTTvoYErbKktWt1i+2v\nSRO6XCJiatYtC/QJKYwQEdTKjI+ERCk3O6umW0lCqlzaYjSaFhtDuw0gPBBDxjwn53SfKM8y8QKh\nIUtQrKogI5EmWd3icbHW5dIA+rqV2ZeJJnYavNKKWMYj4DY1Ja/2WgqMCtdHFFoqTBSxB2532mqX\n2zqw8+WeW7a/YCWbzQDoiPO7BwUHQ7rkkEWLVDIlfoKMYF1K4dRuW6xy0Qj2BJ0Dheq2yyL/i2ta\nBMTyY4wkFsQtNzVDPvu5OSb9Tk8VAYwKoaeVOWzsJmGVHO2WzsggCO/6QZ5o92uys8UoYU7HzygU\na3F266n1SLwacbU4GBU3XlKRvnQyhT0A0ObcIu/mVLEwa91ySnW7je69o0zOqalpAGNFKwUzJgZm\nt26pL7JF+vKCzAlNlEUqjVvChVP7gUUJrX/gfvEWlhcvXrx48eLFixcvXrx48eLFixcvXrx48eLF\nixcvXrx48eLlH6n8PxDy683iIzzpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.Image.Image image mode=RGB size=400x100 at 0x7FC0E80EE668>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataiter = iter(trainloader)\n",
    "images, labels = dataiter.next() # 返回4张图片及标签\n",
    "print(' '.join('%11s'%classes[labels[j]] for j in range(4)))\n",
    "show(tv.utils.make_grid((images+1)/2)).resize((400,100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####   定义网络\n",
    "\n",
    "拷贝上面的LeNet网络，修改self.conv1第一个参数为3通道，因CIFAR-10是3通道彩图。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net (\n",
      "  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear (400 -> 120)\n",
      "  (fc2): Linear (120 -> 84)\n",
      "  (fc3): Linear (84 -> 10)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5) \n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)  \n",
    "        self.fc1   = nn.Linear(16*5*5, 120)  \n",
    "        self.fc2   = nn.Linear(120, 84)\n",
    "        self.fc3   = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x): \n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) \n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2) \n",
    "        x = x.view(x.size()[0], -1) \n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)        \n",
    "        return x\n",
    "\n",
    "\n",
    "net = Net()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  定义损失函数和优化器(loss和optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import optim\n",
    "criterion = nn.CrossEntropyLoss() # 交叉熵损失函数\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###   训练网络\n",
    "\n",
    "所有网络的训练流程都是类似的，不断地执行如下流程：\n",
    "\n",
    "- 输入数据\n",
    "- 前向传播+反向传播\n",
    "- 更新参数\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1,  2000] loss: 2.217\n",
      "[1,  4000] loss: 1.876\n",
      "[1,  6000] loss: 1.683\n",
      "[1,  8000] loss: 1.572\n",
      "[1, 10000] loss: 1.485\n",
      "[1, 12000] loss: 1.435\n",
      "[2,  2000] loss: 1.384\n",
      "[2,  4000] loss: 1.334\n",
      "[2,  6000] loss: 1.302\n",
      "[2,  8000] loss: 1.272\n",
      "[2, 10000] loss: 1.252\n",
      "[2, 12000] loss: 1.234\n",
      "Finished Training\n"
     ]
    }
   ],
   "source": [
    "t.set_num_threads(8)\n",
    "for epoch in range(2):  \n",
    "    \n",
    "    running_loss = 0.0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        \n",
    "        # 输入数据\n",
    "        inputs, labels = data\n",
    "        inputs, labels = Variable(inputs), Variable(labels)\n",
    "        \n",
    "        # 梯度清零\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # forward + backward \n",
    "        outputs = net(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()   \n",
    "        \n",
    "        # 更新参数 \n",
    "        optimizer.step()\n",
    "        \n",
    "        # 打印log信息\n",
    "        running_loss += loss.data[0]\n",
    "        if i % 2000 == 1999: # 每2000个batch打印一下训练状态\n",
    "            print('[%d, %5d] loss: %.3f' \\\n",
    "                  % (epoch+1, i+1, running_loss / 2000))\n",
    "            running_loss = 0.0\n",
    "print('Finished Training')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此处仅训练了2个epoch（遍历完一遍数据集称为一个epoch），来看看网络有没有效果。将测试图片输入到网络中，计算它的label，然后与实际的label进行比较。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "实际的label:       cat     ship     ship    plane\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAABkCAIAAAAnqfEgAAA0bklEQVR4nO19WZMc6XXdqcysvbq6\nem/0ABgAg2UwxAyHo5mhRIkSJdohWrbssOWwFXaEIxzhFz/4wb9DP8ARDlNW2H6wZYclh+RNFmmS\nokmKnN2zYkCgATS6G71UV1fXmpWLH/KcW9Xd1SNRCke45e8+ALezsjK//PKrzHvuci7gxIkTJ06c\nOHHixIkTJ06cOHHixIkTJ06cOHHixImT/78kd3rTb/3TV/RZkimFIACQ87zszzAcZkqUjLhDoZAp\nccKvpEnKg3hxpujbSKOqjh8DyBcG2Z8+An0l1dGiTBlFPGyS2IADjYFbhlJy3DPR0XIaNkcbRzqR\nLtADBxnqWx2eGb2QH/3Gv7uPCdnf3+cAIu6ay02ZzJ9UfrKDpCeV8QYv+5MbvNQ7uUdO8yMlhU0g\nd05T2/tPGKTtuby8fOKj3/rWOrWYE7W/u50pw8EAwLXnrmd/NmbrmZL3OYBC3qdiW7SMgpwWSdTP\nlFo1r6/nAAQ+B+l7PMjBQTNTZmZmuGc+r6NxH1stURJmiq1b/pnj371uj98NuJxKpVKmhCG/G+mX\nUi6VdXyeqDFTmjzs13/zn/EqFm/yKz5/U/WZWqYcDbkUu+19jU2/C93XQMMtB0UAJT/QuHUr7dZp\nQ5zEJ7Yk2jI+rK7R83xMWwC5nP3e7acan9qH3yoWi5lS8Io6dRFArsDJ6e1/lCk//8u/duIgHpw4\nceLknEhwelNoL9iE7y4kCYAiaBl54IMwCE5aTzJZkAu4aWhvG71bgoQfZU9/7Yic7DVEQ51IT/rE\n19j4XooDPpvDUB9Fno4TA8jJOisV9E7WdXmBvZx1RnDnFPZa4Nsg8KY/033fn7r9zyl/NjMtp7fZ\n2CLycgASe5+mGm0qM0qvXDMzJ779Z7ewTkutwjvlpVxswy63JGEPQKnAo1XL3CHQ4W0BFLVKyrqb\nnoY9jG0fro1C3gMnAACCQGaa7DUvd/Lai4IIsuTQ7Y10IkoGI1Itfk8nyMv6MHttNBzqQjRs2RQ4\n4/4mKQcf+XM8SJ4/t9inheXlZWH1O5mSxl2dmscZptxn5CUABpo3/VwQjghoPC3gfo8/c1vSdiEG\nSjyPSpqEADwzeDVvUaQVaE+AnD0lOD9zc7y0YnlGh+WNSLwUQK7I88adGs4QZ2E5ceLk3Ih7YDlx\n4uTcyBRImApMAcPJLTnhskQoz68Ihcm0Nh+fudwKBZp5UZLXR/7kPmZM5uSn9/QYzXk0OFOPBnM/\noWm7vUcbtRPyW53OSJcUA6iVBAQ0ttkKHZ/lEi8w8eRYFXTyZfDndSFhMh3sGAj6DDT0Z5A/zdHG\niMx2Hhvi9kl2IQLmI15yYOAh1i3LnT5jcmrLnyCfMewgx1Mbviv4PH7eiwEUPYF32y5/+bBPx7bv\n876XAt7E0VDQxuB8xC1pLgAQC+EW8vyKIUEIN1l4IZY7otfjGfd3dzNlZZFAJvPH+wWuDF/HtwnM\n6+0fCC0OFWewCMBoZD+uY+Kl3B5rbLGCIXGO11Wa4akXnl3htw4PMqXWI0gMB/zNxrUSgGS2kf05\nI9xtJ/IshjYMNQk8Y6nE2R5PmO5vtvZsBdpBIl1XYmtHS7EQcMmVy4o8wCA5pz1BDCAx++lsx4iz\nsJw4cXJuxD2wnDhxcm5kCiQMElqVlrvhJSNMmO4TURyFbE6F0iJDUhZMUSrK6sqtTGm39gDs7dMI\nzwcFnU4RQCVM9VHJlA/XaaijtMB9PH4U1ggbm+0mgI2nrezPWpEHSba55fIqT7RQM6BhKWC8xoIs\n2/hUOgn3lNX650y/+nMhSp05NnyqZLQoSQCMBLc/vc8kspVV5kkZrl+aJ+QpKUaT/ORD+oxJKCgR\nL4nkSRAQyHsJgLz+9GKuhEJekMSPtWcoRXczJziv5RoNFC70qwAGusCKXAG+BQ4Nt+hKuwPCrjff\nfCtTRkKjc/XXeNiiB0DYDjmD4VohniGd1KLbgqsWxk2mQ8IICpyBizMRQB4qwutLqSrmV6/olr31\no0wJd4kNL7x4C0Bulz+KYY4xx5ou4KjPCGNJwy6mPJq3oLikooQWPB1WSgCCkeDwSEercraLh4eZ\nElx6IVN6jVkOUpg91o0oJbzYXJoC8GLFauMzDSlnYTlx4uTciHtgOXHi5NzIFEhoSCMXNKh4OQhl\nAPCEm0IZ+YUCbdR4nD92MgOzoCDKF//SX86UN7/3fQCbB3vZn10BwCiiRfpwYydT7j95kimluQuZ\ncnH1CrcUWM8RClEWassAogFt472dzUypNOYzZaPzNFMGuqKVGo3hiooD4pCI4Kwn+uko4f+l0pzP\nxIwKbuZVGqW80H5nCKB1SLP/6R6rUsozhAYLKk+xWhMLmVmxztTxjc/6p5OCHAipLi1vZSLxEICv\nMF8uJrjLK1Y7MhAhqOvXDUQo6Vc1NElkML4IoNNuZX/VKoRFnmbSKmYCBYNbCg4221TKSrkMheHC\nUQIgKNh9VxQv5kgi/Rysdq0gV0OqlZbE0z0M4x+dRfFSqyRT0qeAWU7YbZBTjVGscrclQv7eUQhg\ndP8Tjk2ek0QVQd1AF6axFSKVFj1WmnGo+i0Fjge1EgB/wD8DXjGGFzik/pbqn3JL/O7sIi9EJxp5\nFlflVSdpAsCX9yDwzlzzzsJy4sTJuZEpFtbQ42P7sMenchwNAczV+EiuqyInkGfd/KnjysrkZLpH\nr8uckW/+/n/KlKcHAwBPO9zh4SZ3WN98zOOXaGpFPs2oWp1P66BS0z58LZT08C15FQB7MpHWLl7O\nlIFsrvv3aWE1W8rlWePRri5TyQd6t5zhJbXKjDT5CQyOdDxBx7aPE1tOWVixJtXKuX29aa1yYne/\nnSntLq+oP4wBdHsqciryVnb7vFO1iswNjaQwHsyfcBU/kS1ZzFliEWcyr/rYLJdqnEiV6HbkVKPj\nncxj8nOqEZE5ZlNppfgxRgA6R5yTR5axJaPJjKNLdU6LZV298+57mfL5O5/LlMSSwuIQQCm1dELO\nZL8nnKE1E42UPhbw+CNVyA+HPUyTWJZXovy41IwJ/cpCy9XSiWaPNBvLzMwqLz/LMaSH2Zi4w+Iq\nR5tXPfM2K6ihipyuwmLpCiNaeVXRDQSYqjNVAOERr2KoyQnK8pdrBQYLtPVyeZmiKU3FGS0fX4Zb\nlMsDyHlKEsSZdW/OwnLixMm5EffAcuLEybmRKZBwt0d7rBk1MuXb/+vbAG7foGPvl+4Ql81ZsbWl\nooiSwVNFjhU9yLeLBw+ZE9TsFQGkFfrCvZqyP+aPMqWkDI5QaTKh1dnMced6jaN6urWVKe1WE0Bd\nxnBJPtdHAoD5Ou3nna2HmVJ7yjNeqKt8x7MIgDE6HJNuz6gsZGPLtDYuMF88AaYYbZBhQy859s6w\nIiEDZh0hGvO+l+XKHageYkuQcOeASsbTMBLe6x0RDu/I+77xhNP1wo1rmfLclYscrdKIxv5+o9PK\nTfw7Ubrhne2I9+U4TwSUPDkQ+odtABBKSkUJ4Iu2oWCEazaBI4YRYsNWsT4ae/dDAN0uE4KePuWe\n1XpNJxI21EyGHe5TUvhot9XKlLfeJ0isFn0A169xugJB0WGPi6csFpBkyLURKw4QG9YZtDFVLKXO\nuKjGmYz6SLAxL5RdvPcpj/rGdzIlev2L+lYRQJoSkxYEHgfglda2eIG+mCSSqiqWUsVwRvzWzEKD\np36yDwAdLqf8Ct1HeEyAGWiSB7ucN1/em+QmM7MG4njwcubvzwEItFzTs6M+zsJy4sTJuRH3wHLi\nxMm5kWmlOY2rmdLb4+NsVFgC0BRU7IUEWfWCUmDGoTRDQ7RFBxFB1q5M+N22Ig6NBQBzy4zidRJa\ny4sqxLEIYKh8j4Gq0vuKAT2riEZPGHAnHAAIlJbVaurEGmRfBq1f4Imethmg3GoTvzy7KGx7hnXa\n6jNKVauI1zAwFGxsENrbwiKGBMdEesffGaeyura3mIM2P0/sXC7x0oYDXlqlyC2rS0TrGd9xt8fL\nqcoIDwdia9OFdcQ2F42LjRRaGqeA2UeTVzNJDoGzpGRsedrJIGExjQHUFGadNW48pY8VhY9KBo+E\nxD1d+5hmNxYXdjsEMFPl9jnN24MNUjPff0zlk3t/mCmtvVamdAYcWy98P1MCKLuq1wZw5yYpjP/G\nX/tapjyjFTgscbSDLscfdnmieqqkpP4RpkneV1mMJsHChYkcL8ZAWTvg8aMN5hjWC/ylHG3yjGFp\nFkAqwsvcFlMaq89wuYZ14S9wkZQ7Sh9rcZADVU1Fe3QgFAYhgKhNuF1sMnw/6guPlwmZWw8Y6y+U\nCQlnLjCC6SsXLFXi1RApgEgLL0zOxITOwnLixMm5EffAcuLEybmRKZDw1ksMNGz8gHn9tdklAK9/\nidsr3nqmhIqPGBrKiV8tRiNTZoT43nmPEY1agyb0M1fuAEhl0hcMYA5YrNNTZYBnTTs04A/fezdT\nZkvcUlGQsVapAdjcpm1sBRueQOK80gVbB7R4D5pU7m/R1l1bZqJdoFGdkKDOq4iF6UaKkEJBHFMs\n9GPVIYaP0uMppOPooRSrIzEOAMO2DdXZjFQ9DyGLSm0GE5Aw5xv/gTqXlHXLrE+M4rjjGM2pwWSQ\nP3/y88/ChI/X1zVIzuRRm8smHg0BPFHd1YHoIrod4v3lBaK5WpUowlfScmiUhAXx8+n+dgc9AAMb\ntLjkH21yXT3YYKi0F4quQ4FjVHh8IxW3aq3tR58A2BTm+qPvfDdTbt98LlOWGsRH/U6LI1F7m9Ft\nMpR0RLl3QorCdKnuIIw0RZDZk9JRlVvn1c9nSj34KV7REed25Gcs6UZEqQhjmSfqxqK7kCtgJI6E\nvFZyXwz6lsfZj2MAvQ7PUtXRBtqzqJ/h/AxZQGI9HDpaclDyankk/r9cbuJCMTp7OTkLy4kTJ+dG\n3APLiRMn50amQMLKLMHOs9cYEMmKpZ69yoaXiyNihtZ9Jl6OBFLiiGjr9V/4m5ly+dqrmXL1xfVM\nefNtorm52iqAzR0a6oHYvEoKaakmHB3l9R0e0MaeU9dMsx2tFnxxcRHAQFX2e8oAtFKyWk1RSGWH\nhgo53X+8kSnLcwQaNy4qNe64fP1f/Rse1hJHZfrOqEfm9auEw6+99ILOyK9bcmkWiUsNv8g+jzSl\nFuQqFIUajABDWY4Lc8pZtQ5shQIm2AKQl+muoraWAqMtMa4dHbYyZWQ5sQrwLShv8Mb1awDyVqFm\n3TknQOMJ+c73fpApRtVvRZG9QQfA+vYT7UCxWZpT5nBVgdGizpNXKmmgvEdPbb56gxBAoLasqeDw\nVlPE5wrfVmoNnVMEJB2r9eOZrAS1XqsD+OlXX8r+7B42tQNx96NHnNJ79+7xIwXPH+5zSvsKIJ6Q\napXrLdKVjmK7C0RzRpeSEwour3B+2urqunvI0eZ8H0CoZmUFC8C1uGck5F9Ujndba7JkHQ2MLlE+\njWFWnaq2DId9zZvwa0V1jjMXL2WKbx6GcWc53eBxPnIKjNdTcnbmqLOwnDhxcm5kioXlF+k2e7L9\nYaZ84dXXAFRnaZj4R8biIONCr9wfb9Ab93NzTOZChQUfM1VVPwQ8frlQwUQxhPmS19bo8P7wHot4\nCnJJtuVTvHqJ1t/N51lV32yqg0g9B2Bzm4knnt4SDfFhtcSUZDZXudLIlL7K0D99pPKgwvRn+kD+\n7LCv8nSZM51DXbq2xLef57dSI/YVL22hjAlTZUx2LFNrdp4pPGMiB+t3YvwNskmtACrhvzzausqh\nnuxwWpr7tFX7fdWRDPW2FKODUQtcvESf9OVLFwFUC7ZsLHRwpoX1zl2euqISDbvRg6gHoDHP3DG7\ny6GMmp2O5laXXCtx7UUyFT1rrarWSl5QBVDoqhvoiC78ZrN5Yth2a0MRRbR1RqsGu7zIZTM/fwET\nFT/7B5zJhQbP++rnuRQ3NmmntwecqI9UuXKaTJwXKD96eYYX2FHKYaBVGltClipaPC2nRMliOV+x\nCM/HhLN8JPKSstomGbwwW9V87bHm1nrwRCqJy5dzABKlvBnJnfE65CM1NrZMQ323FNsql+VpnNXI\nYeJ25M5eTs7CcuLEybkR98By4sTJuZEpkDBfordyMDBoMAKQVyFLpWquUDr/iqJbnQloQ/7WP/8X\nmfKrf/ef8LAqUygUzYaPAFy99kz2506TzteBHJ+rywQLRlw7VCuUa9cZAXjuOrHh4dvsd9I96gBo\ni/Q2iszFSyO/oXyZpEWre3ZO3Axy1fseL2RjcwfT5O/8rV/jkOSirp7qE1kWdDLO4XZbbAoigcgH\nJQCB8llS2ed9ZS2liXLQhCby8u4HZsznrdDnGKK0fJaBaA+MsWCu0ciUWCyAJZ/jb+0T9Ww8Wc+U\n64q3+F6ACdzqC6V+RmnOkSXCmatb2LDilQBcvMQ8pnDIkext8yt7TQZkVpfJBldapCu32drXUblz\nfY64tVScAzAQy0Yv4pyXKrrvEe/7uLerHPbm3Ih6HO3rP3UnU24+uwZgENJr/uDH/Mq9Tz7IlJ95\n7cVMuXSZS/rRe4xKmb88OaPopKBsr4LyChPR3ZUVMInEgHjUVutTEYSUZolbV6qKEaUJjrUsFQOi\nbBRf3oNxZOaUpCoPMkgY+ykmGBA9KQVDnzrsUOSLRtMS6NpjTfu49VQSYKJwzSgqT4uzsJw4cXJu\nxD2wnDhxcm5kijWYU3FAT9Bs0OsDyKu95dG+akRUiJMHQcSFBi3DTz9kKsrmBhX0iPgePl7PlC+s\nfhHAM88yJri2Q6V7jzvMFxqZUm8QG96//4AnWqPV3RLIGsl8fbqzDyAxqgRZvD3F9bxTDAxW1mMB\nrIJ4zsL9bUyTRMloYxtbH9UKrJgplzhjfdG29UacuvX7vMZCoQzg8lUWsj94zPr73/uv38iUSNGc\nko5WMUVAslEn2GnMEhF84QsvAVhaZHnEcxc5XV5OnIKy1C0SZGGj/jLxxdqFBpVn2Kwo45DrKbtn\njILPfvHlRcy/uMwxWOB1b+8xgE5XBAYqzbBksYaYyNeu3ciU+iyvqL5IkLin6HAi7JxVofTUKLSn\ncFsYKrNJJAQFY3kMeMsKYmpfXuWULs1RKeU9AEsCnnWlL+0/JO57+OP1TFlV3LO1/X0edp6jDc/A\nX4F4C3w1iC3pZ9jaYXCz2SFlwu4Wo5BzM0yZvPMC0ai1K874D0aKx1lU2parNSUwV0NuDPC5czwO\nR1o8L/vIvqtqm/F31VBHZ7QlZzvnlRmXt2BgCgCeEG58dlqfs7CcOHFybsQ9sJw4cXJuZJqBarUm\nCg1cWFzABBL55rtEeXMKAN2Yp7FXKljYhfhrd4cgLhm2MuXydVJ8+aUigEqdRv7iClNM91VC0VJw\n0PgBl5eZRRkIn1oJjpXvZ9FAK22xDMOBIoyRWMYXBSvgqQmryMxKinFEYhM8Ib/7e3/AsYn32lPy\nXk3h1BkhtSs3eGlLC8RHCxdYtTO/uAygJDaC1kfEF+9//ChT+sKvBibsvsyIrv765SuZ8qUvvsLj\nV2cAVH3V0MjEDjVdkdpk9awiRw1ByzpsoyG+/G02RtvbawIoq45kZZUTWKko+/eUNATnJwqhRMIH\nD0BT5HnttlIljfNbKO/RBgdQb/O79dkGdxYdXE/5rshFmKwvqfB2lCpWxGMAhzNZ0z6B2pdeWuC1\nG1tDt90CEAlgGp/9VcHVjz76cabcvPW8js/Z3lQqaUmFVifE4Jh1BkiE1I6ULL27S+/EQZNH++S9\nH2bKx+8Se16/ziKwK9dvA5hbFAuFQJaxSxpPv6Ev3+hGtC0Y9yI41mtuoh2sgo/a08LFpzsNm4yD\nj2POkuws9lOd3lsPzsJy4sTJOZJpeVh6WNZretPOlDHRUrQtsqC9Fp93i3UepypPZKQOKOvK5VmZ\nb2TKs3oJZJkyP3zzo+zPJ1v0ns7UaHPllYHywaePNDorPVG6hx7GnS7fvXML8wAi7bCtGp1anQMI\n5HSv6L1qRSEImfiT9DiY1eXpxc8/evt/Z0pZNEzDkJ71vJzKP/3Tr2fKwyeki92n2xR3PscyjkK5\nBKA3pHWQlxn7yiukOhqoGap5iG9cY9nT58SytLbIS6tXaPskgxDA4232B905EAf0Hrd0O/RJt1Qc\nHo7UKV4nsnJrq8EajSIAlQbn5A54FbOz02cJQHC8JhkTL8msXNnCI4FqtmxLocTDLi7R61+r8QJL\nCjgEGmSQ543IctBSFYJY36NZ5aB51u1JnFCB1bgMlZqnMus04rTE8RBAqNKTvi6nMsO0xIfb9I5/\ncJ/Wt9X3jFT2lLbPTHrKxEyVkvjBn5e9dv02oxa9I973D95i7uFbb9DC+s531jPlww/fB3Dr9svZ\nnzdu3c6UxlwjU2w5+f5Jw0qVXZNbtACSGBNZhCZWrBPLmE/GKWBnypgVLudjooouSs7M63MWlhMn\nTs6NuAeWEydOzo1MY2uQg+3C8gXt5AFIlLBz4SIhyY821zOlBTVrCegmbyzSLTdbp6Fu+ThXBAlr\nswsA/uVv/uvsz56O3+6z6qKn8hrzN682RJXV5Km7RTsRvaQffbwJYEe0BObKbXgcZL0haCAWpCCk\nMR/0mAY1XxGOKE03aXcfE6XOy8a+eJEe6Bc+z2qhvGDF++/8MccvO78mkqOdvS0A1TphxUKdO/z1\nr/08B6kcp9lZ7rO4wOybZpMT9eAh6acPW4Sl7cMjAEeKWhyIhqmpfieRQhAFebgLgvNGYlGvc/xW\nxzO3PAOgaFC6LGoBUVaclnmRTSehebh5oiTqAyiIZWF5ZS1Tcqo9KiiryMBpSZUrvgZptBbG/pzl\nBFmiWa+rQhxjgJI/PhU27B1yJp+scyabyhFqqKvrykIDQEl0EeYYTgOi+EClP3tqZnNxjUuupmtv\nD6a7k61kx1oRp55tkWNbmVmNBdYn/dxXuOSuX+dP8rvf+lam3F/fANB7WywUYih58SW6Gi5d4kEC\nRWbiyBi9rZBI12jO9DTFRD9gIxCx5k/GdTXuA2ttay29y+qTxk53D0CSnsSVp8VZWE6cODk34h5Y\nTpw4OTcyBRIa8W59jsZ8FAcAijJ9b4r590dvEFsdFljNn4Dm98pFHvmDDxm/+Nmv/MNM+b44c7vd\nNoCRAnM7WydDgZ1I8SNht4ZH7PZMmdjncJc2fOQ3MmV1pYEJa9YqcgbiQe70xMWsjqrRgGVDywFD\njWuiUR5GVs9xTJ7cZY1+W7GnX/3lf5wpX/vaVzPlD7/JaNGy+CGW1XW1rFSgUi4BsCI+3xkpJSVD\nRbLGDRZFSmPZ/oTDfrTDNKVQ7XOCUhXAzAyzfpYFZEbhyfhOXkjQSuRNmZlhkK5en9FHOQAdEfI+\nfcp7Z3N7WioCSpGnsJqSzhr1ZQDJmAaS96Vc4+lSq+oQbElSbTlFs5uaggRApBsXxRxbe19k3Hbt\ngoSdQwZPN9XCZ3Ve1U5Vpv5lPZwSQdFIh7Fw5DMCWbduMtPw5Reo3L3PMPHb732EaZITEvTEZeyJ\n+CTvW6GMsqIUxfMUGL1xk8TNiX4ym9v/AUBzj+A0UXHY0ycfZ8pzNxg3vP05fnd5RS4g/dKjkfia\nlcwYpzEm7ssUamzh7tMkfGOWx/HF2pdSYIwwxxU/p8RZWE6cODk34h5YTpw4OTcyBRJWa4Qtc4uM\ncWQ97weqXynVZC0rePToEYsGvvw62c4GHSVn1mlsb22wnuDTu3d52CjEuDEHOuJdqM8TilppTkMp\nrLdu8fg/fIeW7Vsfr/PUX/mVTMmIBu/f+/TEQSzXdKDqisurRHMl5VvOzwuMiJIwCqfnsA16jLu9\n+HkWyv/SV38pUxbUKfZnv6hIn6DHjCqK6ppkv1DCRDdQi1sZS7c1CqrLUE9EDHFNs7F8kXHJ5gHn\ncKbRADASWskJLxlvt4WlrOlLR9G0VC1SjFb88RYTXgf9HoCRUHasEo1K9czSHIPktQrn1vDdzu4+\ngLYyVy1f9PotJkYa3bufNzRExXBxqIYtPVHr9Yc9AJHyeD2VHCVD7lkTCjaa/3JBJV+KfzXkE5gV\nyXo4HALoaZBGN+ipoGROcL4iisqNxyy0EqrD556/gWlihP3+WJErwOqIrHQmORZcAxAK6V+8dCVT\nrly5CuANaycszsKdnRYVocWPPmIXq6tXObbnnqOyssJU1RklxyKXBzBQW9ZYv4684LyFAi1x1Cpz\nUuOxHIutzxwmi4Qcp7sTJ07+Aoh7YDlx4uTcyBRImETEULPzREzdfgygJ3xhUaTLl0lCcPd9orzD\nnpIDq4wkXiZhN9Y/Wc+UzU3ii5/50msAusId9TXmDc6vMbbyqEnc11NL1UKVNvzsMo//Sp1j2N1j\nDGh9fQNAR7X7LbWWXF6i2V8HjeErNWK35brI0UEcESrGVD2DS+za8y9nyq//g3/EQcYEGp/cY8wu\nyYnEQpHEkTLimi3Vuyc9ALG6ZipGhATEL0dtFuv7T2n2byondihUkigdsaoo5P1PNwA8eKTAq1Ix\nFxYX9F2l6aqR6p4mEEogHDMdSqlVygAaJZ7FOAX7nemxVEzkozb3OOz7YmqPkiGARoOlo2trpBYI\nVao2Cgknk5RDaguJ9/rG5DHUaIWh8h4mcF9J3BJl5YuaTyBRuK0qBkdDZAVV2Nlqz8KpRi6Y80/G\n7Eai4d/YZ+Vmr9vKFCuoXL1wEdPEF1wyBToRcgrsjtMsT9X66SOrQKzP1IHJuk0pRrGvyHu7yfvy\n9h7x4wfv/ihT5lX/u7rKn9vq2hUApZLynBcYWFxaoRvH0nftlkXyMFjr1nHiqOWdJh4mWBzSM5jv\n4SwsJ06cnCOZYmEdiVKgLA/xcBBCnS0wkZi/NM/X9V15zneafAHu6Z3cmOGj9/aLdEnef8iclIzA\nypziN2/Qc3zjKq2y9c1WpmSl5wD294xfQd1f9G7ceJ/m2NZ+G0BOIQJfFf+rF2m4XdFT+rJ8+cZ+\nNZQplyR5DXJ6LcWv/f2/xwGs8p357vuMKpgHNBy3CVG9hVy25lbM+prE9m6xHp/jVwm3hHo37u3R\ngrNUI7OEGmKJylzRzX01Rpe/dm9PjULFFxzJ6W7FOtY5pqK26SUlH3mRDyC0jjRqf1JWatVpaal+\naGuThm1VlT3Pv/AigAWxklUU+hiI3fjggGl3xiTRE61CRXlqs3Wu0qp61pcLeQCBbKVYTvcsyANg\nJKLqgXV2GXP+iqVXeEKZbQj8AoBULVcHQyr7uzQY9/YZX7JqMGPCMF6QokiNT0guNQuLW8xFnZOp\nYtwGExUxVMzn3e/QHt/e2gKwtUWjqX2oCjkZhjMK+9RklJXEO2IUchvbXNJ319kNdzCIAUQxD7K4\nRFR05w7r7W7eYDLa0hJva32WkZNimU+AFFot+oHQpjfabud0d+LEyV8AcQ8sJ06cnBuZAgnv36P5\nd1nJ+yUvBJAIRARmQ5qHT07lmkiBn3+eqTR/+Af/JVN6LVqnlQX6Vu9t7AC4eJH+vKu3SO9bFCR5\n7ln2kmk1W5ny4Yf07icCIxsHtPPbfdn5cRFAu0WkubxKG/VRk1vmLzYyZV+QB+qV0pK/OVVDoEEy\nxDR55503MuW9997JFE+GrifT2koczOcKGCMCjeqg4GFiJgtjogLx+SpFy0v5Ub1IL7UnAozIt2tX\n+lgKAAUhkUgsgL2WRRV0XVasIxQaynsdqW1SRzioUggALIvALxAuK5xZSoH5Zd7ueWEEYwHOFtKR\nCqSOOup4WsxraOLVkxv+mRVGToq6d771jlUxVnfQBzBQsKIlXGmQbTDgGW/fJjdeXhmFE3zBJ1v4\nDLtHADa26dDY2aWvOhSUNnIRyywryFXS0TV+4xvfwFRRMldiOVaR6mOEFq0PVM5X0pMglS83/Ltv\nvckztnYALCiJ7PEWr72uZLG8VrgF2epKPQvEjlIITnpgOl4HwH6LgZr1ByxQax1wWt56QwtYpJiX\nL9MVsyZa8Atr/EmurXBLtTYHIFcW5YN3Zlqfs7CcOHFybsQ9sJw4cXJuZAokfOdTBqEu3yEleYIu\ngJzFy2S1ttXPo9VioGRh/uVM+ZWv/WKmvPx5Wt2//R9/J1NyKvWenZ0D8Mwao2zGue5HDBLNr3J4\na9eICFrCIG+/806mbHXEvZ2nrTu7ughg8Tr/9AXHYpnUd9UI59620rv03LY6la6uNUpsir6FCfmj\nb/+PTOmJGq2Q52HLqkGx6fVTVfZbG8u8QcIcgFLxJMouiF8hUOpZSW1lDWgodgevZC8ehV3CEMBA\nZTEjBcgs88heVcF4i65UkHy2RujRqHBLreIDKAT8Sl4pQrl4OnDGRGcUS9oKBHuTDOxYDYoyniz1\nrSTcN+hy/P1DLrm+uq8GBZtSEcXFEYBPPiRaebi+zpEI+BuSWrvANKJ5kSP2BetMOZA7otnaB9AL\nLf/L6EC4xXr62s2oCFttqbZpW7UyJ2QkhG4h5lwk2gZDi9o5VQqVhRQ7Cg4O+jzOrZsvAHjl5dey\nP994jy0I/vhHzLE6FKl/rLWxvMqQ35e//OVMCXTL1tUs9vs/+D6Az71ALv+6XEA7uq5tJQlaTHZV\nJBBXr17hGRUT7x4d6opSAHm1sx2c4hQxcRaWEydOzo24B5YTJ07OjUyBhHfbBCN7sagL8gMAXij7\nLRH/lrLs1tRQ88tfYqSvlGfc6uqzLPj+q3/71zPl3//Of86U3e1DAJuHRhvA/qwFWbzNHpV7YoOA\nIjLpIpHm3DJHa2AnSxlNSrZdJGRCsoeRijbGiZG0rbsezfuR4l5pMt06XVmiMbzVZ/wljluZUlez\nzEClOe091moctWmHj2KLfw0xtRZBHGaFMufWMG805njj+6ai1q1VkazHWVauHVb8ATkBqJJwX1mT\nMK8U3EtSLq4xJCcgjsHgCICndrOBMEmjXj45fsndTz7MlBfuEEfYtGej8xSaS1TDYbCiJwb6QY8R\nagsXWqPca6IzX15mgmJW+REoVjsr9kQ7ryXlWvLnRx9/kilGUGHOAcuizBZYR0mhfXEWGiQM1avN\nOOMfPeXasAzS+IwGVuO2o2P2dP5vJHlCzEgEEi2oWVY4+Mtf+ao+8TDB137zZbp3XvwpKgqujuff\negVcu8bM7UAzduUGSf7WLt8CUC7zdlufARu/9Rkw3Le8xNRxo3zwhZQ9eWniZAhgpCtNctNnCc7C\ncuLEyTmSKRbWJ2qP+rvfpaPuC88uAlgtqHm3XiAXVvnsvLDIl9hz11TbqRKKrV0+cb/+b2lYvfU2\nX7lZxc9E6YucpnLXxSUeNjY3s/zl1ic18tSI/PilDELrPmJ9t2kn+LI7UtUMR7LO8lY6Y0lJ4fQq\ngXSkEvEq30JH1jUz5kv4+du0KZILtLl29zgbO6Lr7bRiTLylYyVSJRGPVg34Xnr+JfJQb8q5uyt/\nfz88+drPSn+KKq6q6pY1qpyuJTX7WV3jTbyu2uOVEqeuo8SofdXHZh7uquIAtRm+aRcW5nCGjJT0\nNOhwtJ6sy8yKMHos63h67y7tnSMLaOidnFdQwhrfJ1aqbWW9cQpgcYGDNHuqN7aeqDx69PjEPqZY\np/ieCrAPWy0A3T21yw1s2Lwco+jqKk0pUo1RPO7tPt126PdpQvpKHwtEBh3qpxQp9zDSldphjd3M\nqneiOMJEM5tQ1uva5au6QhWHSfFEmvbgETPX+qGhFrFmz16dPN3BofpOaTaq9Su6UNX5H/LSNp82\nNVqOsqj6uayyKFdTdfrBmU2YnIXlxImTcyPugeXEiZNzI1MgYUd22jfeYh3Mp/fuA/grr7Ig+7k1\ngpQH90lD/POvkau3lKer+EiI7Lf/G/M+3v6Axfq9SAUxQQmAJzfwuJeknO6pZz45fjQUZBtpS07J\nNUMdNjM3g+AkuKtUZH/KtI4NQ2ge7ESR2mQWlB12QvY3Wcgej2i+9mXt96zHqjpfLolAKj8kZCuL\nYKHvpwDS1ICxsIP8jr0+weOXXyfAvHObpMyPHjE7Zu+ATn0rE8kc2oabSjrdkiBVQ8xZsc64vcej\nfSJeJMjnWl8mvKrM1gFUZvjdebFr1eR8PS1l3YhQiMxCHFnQxpMzuSDcak16LDJQk1PZU2ZQRRcS\nKWfn7sek62g39wG0VA2TWLtcHS3QkiiJ5MD4LnrC9bsi7TJI6HsBgDmth1B79pQSFokEIhkDwJO0\nCrncdBPh29/+nxx8RMLiipKSElE/G/nHGIQmlhopwmgLESQRAE9IbSBwl4z5sKz+RlGXBmMstZqu\nUT147Dv00Ht2B5UEZwmGenpY0MMbE4+cuvZxUmAMAFUdRIGs0+IsLCdOnJwbcQ8sJ06cnBuZAgkX\nFmkZNptEJVutAwDfU6OaePSs9qXVtyQSO/i02H/4Bin3fv+b38uUYVLVOcVq4B17XMaWYyVD0Zqh\nGieslddYjCZnBSXGkeD5mMj1mDH227H5OjpxtEQkCmZar64S49TrVN7AMVlV4G/jEbFhNLTsGCoP\nFO06VJ6UXXBX6V3daAQgiQ0SiodaIGI4IOJ467v/PVN+scoruqMr6os+wUJmWR3VwCJcAhE7exzt\nQxEW7/UY9hoIHpUEAOeVXlea5fj9cgEChgCKwpU5f8pC4iVrkAZGPNVmZaMd6AItx6pkeTpSLMAX\nNulYeGw0x8ZZbKHeoICJkqygZEfjkELh/Y4oPSxuaB1hLTZc0vhH/RDASIwCFpC1AJ/5NCxzKlKi\nYhob7J0edC5phURCgoFqwoLCrEaiuKQ5T8b83epVYyCRa83CiOHx7ZOXaI2UbA9ROap3VDhQ6dXx\n32ykJrgGzC030NNoPZzEjyZhx259BGCgz0vBHs4QZ2E5ceLk3Ih7YDlx4uTcyBRLPpAdmy+IQmxQ\nBPBgx+okmPn5C6+Qpa/cYEF2W5zov/HHhFAD2agjGfxFMXtlJrTlTJr4MiZPW8/F00jQdlYtTrlU\nxkQmm5GyH6nhipVHDAVSZhus6li9QKUmHNETI8UJuXyTJGTtLiFVd8PsWHG/Ceg1daKCqmpChQXj\njLHbYLAdIrW4Erfce4/x1sdHnMklT+1XlS4Yy+rueAmA7ZRo5Z6ikxtiBegZAcNl1uivXiGbWkm1\nLDCgJ8q9Wq0GoKIonqfE1PSM4BeAtpg8ekoc3dkUB8MgBBArRdYoJUaCbHZd1kI0L6KIcRRYit3x\nbMKMm2HQURxZRAtHh1QsNludUVKxJjAdKTAtFsMsr/VQ3YYMCcbKyTRi+OTU3TSCitwYsh0TyxPu\ndBjwrcjFYceyTsAWCgwjG5syLT2LG0YAQmPpEPeDJZ2Ow4WG2cdI08alORT+zb5lZXATFWXxCcUa\nqXqnfsf2USAgORpFAHpzXFcXLs7gDHEWlhMnTs6NuAeWEydOzo1MbaRqPT5lKwYlAGFEu3ynQ/vz\nzY8ZsvmVHm28o5QAarNJpaQgXdSzHDYRhFcqAAIZq1Yfn9OoLKxgMcFUHAbGhGfFZZ2Qww6jLgQM\nMVH+bgCwO6ChWxMSnFtmPZ2RiH/8gCHRfGK27jGpNxhKW1phKG1LkNAsYKvMH8pOtp5RsXo3xTgJ\nH04M2w43Egbp7jGtzis2MsUX68CmTvQOhgDuCUB1apy32kUW/S2qbe2irr2o5MwxF5+gTVEM9H7g\nA/CtyaiF87TltGw//FQHO1kBl0XTAjG4Ww/OnHUzzRMWVUSOmDuFXyIVhHYiRRKHEYBElXFjAjz1\n+yoUGYlbfoaT0BVcbR9QsaZn6TgKmcMEgZ+xOKTpyTtl2DBvRAu6y73edA/DxiPSDX66zfNWldQa\nCEVG45XFGYv1UaKgc36chj2CKgoB6NLHLgZrEGtd+8YxR9tH99dmO2OkSOKT8VBPvo6cGErG5PRa\nRafmCR3l9MZzFQDPvMQmEnUlFJwWZ2E5ceLk3Mi09JkxZY96cnh5AIn1mJSZs77D18XXf5utcb76\nlVcz5f4mrYDeONdJvnxVV/iFAoCK3pkF2Up9FVWYvzyVcZQv8dS+3vnmoLUt2aO9b3k6uhzboSHj\naGGVsYLdPdaRt1SV0npEu+D6NVW3H5eSOtYUdTn2covlr5XfHFHu5JSOi/azneztY/vpLZdK6egt\n95Fe8rNqqPPxgKzW74tdulmvAJi/xMGvXSErWWON116ocPzWhHWksQWya3wpgd722Rt1bCLl7AV7\n5pvPT5SmFJu7VzZLdjRL2EntLc3vDsW8HIkbI9GcTvAfUMzpnnUVNesgkKkVaxWViqpYUu3RwR7t\nmq5iLHmtdt+6ew6HmOhhYybweBK0kq3jaUlLriPaiV73ENPEg1aRLYTYiKRlAdkk+5pJGVDWHtWK\nsbI5tilNlftmk5sadDCCCuvBI/7uWJ+NzJTz8wBS61Rk5F1mnVnb1/H8KI6h8EgkMuu6mEIuvngT\nQJDj7WjdfR9niLOwnDhxcm7EPbCcOHFybmQKJJxXU0mrmehGIYCCOi9aKoenRK3v/PC9TFnfpBv+\nsEcbuykPvTJCUBUYySoMijqI4Y6S/OW+zHL7yGzUSEAvZ749mbhxOMJEBkpZSHNxntQC80tEgqHA\nwlB1/H1j7xWgyLpynhYrlO+qWH+mwRMNugQyxv0QyyqOxwa/xm8W9HFJhX1SJUN1lWLzXXFVPxKF\n9H5FuUgrzA5bvbgE4NoiowoLCi94mvyuAKDVQwTywtaMOVo7B0qdK5UrAIqa0rzIOT5DzNU9ZgFW\n+lOa5ACkikSMkaa+ay722Nz8QqnFohwLWiTG+pDy4OZv1u1Q1CJU+lhP6UXd4zUiAHIFHnagPMFs\n/FoyY0xvkNC2GBtEGvLUB/vE7KPwjOVkpJXaYSSsbh9BxTqWg5gIf3maWyPqS9IIkzBcnpmCrt3w\nZZIeQ+iYyMMajeSrNy97mmKiFe6YhcI8C6nc/7mTP9WRqC7nbpKC+eJVlvQNnu4A+LH4NsoipDwt\nzsJy4sTJuRH3wHLixMm5kSmQcCgQpE4rGCYjAHlxM0RmFRv/gUJm65vkALA6+yg0y9a646hZaa+D\nidiKlexUraFLhdjQk8FZMP42oRWrvN9tijEaESZKN+aU1LG60KCyykhZS9it3WI9REfdTRrqfLO3\nM71wPFQxhF+gxTu3xBON1H80UrhwZJE4494WJMyuzDJ3cqeCg1BVRyDeu1FZpS2zHORzsysaNqtq\navUAwIygYlGVRgOr6rB4pdHaiT9vjBY0hrwgeRZpzWtPS8hKz6AqBzAIrfTfIlbH0nw8XaCRu9uS\nmIB7wiCWPWSwKzm5wLJamZHSCX2t55FwX6zDVrUUDQl6RpLRV7HL8T43yal4riVkBQLINi3Np/w5\njIaM3uZOQn+JBQBF5+ApXphXTA3x+IfHnRV5H5M2mIshzQEoCdg26mK4twvRsM25YT+Zgm732Pmj\n72WRRPtK58hi8Tqs7mbbgs6LPPXlmzczZX6eDoqNj9goa//efUxknJUKZ02Ts7CcOHFyfsQ9sJw4\ncXJuZCokpDFcFAbJCvsTxS9yFqSwou2xchIJpslJ0z0dl3onmLD/Dw6I6ZpiSa/XWJAxK4BW92hM\nJurtGSVDXYkgQMkHMBTlmDGIB7KWo566MPW4T6fF7luJyAyM7XsQTH+mB0oTbSwQnNbESB0PxWom\nKGgNoNIxmZlRC3iYQCLWm9aI0AJBzrKyEOs1VZbUSO1WE1NFTbA6s+pDcfJ1NNqeAQHjNRcrQEGI\nzACgAbEx/kpTAKGK7AsFKUo1PC35ovE1KnPYPAmehwmmh3FwcJxmezKwCEUSLQJrlWSRQloZsX1f\nSDDuq5hGUcKqvlKeZeDY+OdGKtvyToE3onWL/I4bnlKrCq522/QwtJUvaojZ7jtUlcLt5mcx8nWV\nSKXim/RVkWOKDXLM22d1NrkUE5yIvaCtAYxBob4rX42mJQjtbnqnvnVMIo3Nzmvp5fVlFYHduqZj\n8UQf//AHmTLc4e/Oj2NMVAslZ7SbhbOwnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJEydOnDhx4sSJ\nEyf/z8r/Ab+8NWulkQjIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.Image.Image image mode=RGB size=400x100 at 0x7FC0E80FDB38>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataiter = iter(testloader)\n",
    "images, labels = dataiter.next() # 一个batch返回4张图片\n",
    "print('实际的label: ', ' '.join(\\\n",
    "            '%08s'%classes[labels[j]] for j in range(4)))\n",
    "show(tv.utils.make_grid(images / 2 - 0.5)).resize((400,100))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着计算网络预测的label："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "预测结果:    cat  ship  ship  ship\n"
     ]
    }
   ],
   "source": [
    "# 计算图片在每个类别上的分数\n",
    "outputs = net(Variable(images))\n",
    "# 得分最高的那个类\n",
    "_, predicted = t.max(outputs.data, 1)\n",
    "\n",
    "print('预测结果: ', ' '.join('%5s'\\\n",
    "            % classes[predicted[j]] for j in range(4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "已经可以看出效果，准确率50%，但这只是一部分的图片，再来看看在整个测试集上的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000张测试集中的准确率为: 56 %\n"
     ]
    }
   ],
   "source": [
    "correct = 0 # 预测正确的图片数\n",
    "total = 0 # 总共的图片数\n",
    "for data in testloader:\n",
    "    images, labels = data\n",
    "    outputs = net(Variable(images))\n",
    "    _, predicted = t.max(outputs.data, 1)\n",
    "    total += labels.size(0)\n",
    "    correct += (predicted == labels).sum()\n",
    "\n",
    "print('10000张测试集中的准确率为: %d %%' % (100 * correct / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练的准确率远比随机猜测(准确率10%)好，证明网络确实学到了东西。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  在GPU训练\n",
    "就像之前把Tensor从CPU转到GPU一样，模型也可以类似地从CPU转到GPU。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "if t.cuda.is_available():\n",
    "    net.cuda()\n",
    "    images = images.cuda()\n",
    "    labels = labels.cuda()\n",
    "    output = net(Variable(images))\n",
    "    loss= criterion(output,Variable(labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果发现在GPU上并没有比CPU提速很多，实际上是因为网络比较小，GPU没有完全发挥自己的真正实力。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对PyTorch的基础介绍至此结束。总结一下，本节主要包含以下内容。\n",
    "\n",
    "1. Tensor: 类似Numpy数组的数据结构，与Numpy接口类似，可方便地互相转换。\n",
    "2. autograd/Variable: Variable封装了Tensor，并提供自动求导功能。\n",
    "3. nn: 专门为神经网络设计的接口，提供了很多有用的功能(神经网络层，损失函数，优化器等)。\n",
    "4. 神经网络训练: 以CIFAR-10分类为例演示了神经网络的训练流程，包括数据加载、网络搭建、训练及测试。\n",
    "\n",
    "通过本节的学习，相信读者可以体会出PyTorch具有接口简单、使用灵活等特点。从下一章开始，本书将深入系统地讲解PyTorch的各部分知识。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
